The evolution of project portfolio selection methods: from incremental to radical innovation

Abstract

Purpose

The purpose of this paper is to identify how project portfolio selection (PPS) methods have evolved and which approaches are more suitable for radical innovation projects. This paper addressed the following research question: how have the selection approaches evolved to better fit within radical innovation conditions? The current literature offers a number of selection approaches with different and, in some cases, conflicting nature. Therefore, there is a lack of understanding regarding when and how to use these approaches in order to select a specific type of innovation projects (from incremental to more radical ones).

Design/methodology/approach

Given the nature of the research question, the authors perform a systematic literature review method and analyze 48 portfolio selection approaches. The authors then classified and characterized these articles in order to identify techniques, tools, required data and types of examined projects, among other aspects.

Findings

The authors identify four key features related to the selection of radical innovation projects: dynamism, interdependency management, uncertainty treatment and required input data. Based on the content analysis, the authors identified that approaches based on different sources and nature of data are more appropriated for uncertain conditions, such as behavioral methods, information gap theory, real options and integrated approaches.

Originality/value

The research provides a comprehensive framework about PPS methods and how they have been evolving over time. This portfolio selection framework considers the particular aspects of incremental and radical innovation projects. The authors hope that the framework contributes to reinvigorating the literature on selection approaches for innovation projects.

Citation

Flechas Chaparro, X.A., de Vasconcelos Gomes, L.A. and Tromboni de Souza Nascimento, P. (2019), "The evolution of project portfolio selection methods: from incremental to radical innovation", Revista de Gestão, Vol. 26 No. 3, pp. 212-236. https://doi.org/10.1108/REGE-10-2018-0096

1. Introduction

There are two ways for a business to succeed at new products: doing projects right, and doing the right projects.

(Cooper, Edgett, & Kleinschmidt 2002)

Project portfolio selection (PPS) is an iterative process whereby managers select projects from available proposals and current projects, in order to meet organizational objectives (Archer & Ghasemzadeh, 1999). This process is crucial to maintain the competitive advantage, enabling the company to focus on the most relevant and strategic projects. However, firms struggle in managing this process. PPS’ complexity relies on two main facts. First, the information required to determine whether a project could be successful or not is extremely difficult – if not impossible – to know (Baker, 1974). And second, because the project and environment conditions are too volatile, strategic goals might change, promoting high levels of uncertainty (Rice, O’Connor, & Pierantozzi, 2008).

Different methods and approaches have been proposed to address the PPS (Bard, Balachandra, & Kaufmann, 1988; Cooper, 1993; Regan, Ben-Haim, Wilson, Lundberg, Andelman, & Burgman, 2005). Current literature offers a number of increasingly sophisticated project selection approaches, including multi-criteria decision methods, scoring-based and simulation techniques, among others (Zhang, Yang, Dou, & Jiang, 2016). However, the current literature presents an “interesting dichotomy”: although there are an increasing number of frameworks supporting PPS, the adoption by firms and its impact remains limited. In addition, there is a lack of theoretical foundations for these frameworks (Kavadias & Chao, 2008).

Other scholars have also identified similar problems in PPS literature (Beaujon, Marin, & McDonald, 2001; Cooper, Edgett, & Kleinschmidt, 1999; Kavadias & Chao, 2008). Indeed, years of research provided a fragmented and disperse knowledge on PPS, increasing the difficulty of convergence and accumulation of knowledge. To some extent, the existence of this fragmentation might also impact the adoption by firms. The development and adoption of PPS are more difficult for radical innovation, in which managers face high uncertainties in structuring and selecting projects. According to Brasil, Salerno and Gomes (2018), inappropriate selection methods might restrain radical innovation proposals, since, typically, the PPS process has been centered on the assessment of ex ante information, such as estimated costs, profits or market share, that only can be confirmed after launching the project (Zhang et al., 2016). In sum, there are two major concerns in PPS literature, which can render the consolidation of the field more difficult. First, there is a lack of a comprehensive framework regarding how the PPS approaches evolved. And second, there is a lack of understanding of how these different approaches might be associated with different degrees of innovation.

To reduce these two major gaps, the following research question drives our research:

RQ1.

How have the PPS approaches evolved to better fit within radical innovation?

Employing a systematic literature review, we aim to identify which approaches are more suitable for radical innovation projects and what aspects are crucial to manage radical innovation portfolios. Our goal is to go beyond the last review on PPS methods, dated from more than 10 years ago (Cooper et al., 1999), capitalizing on more recent works. Addressing this research question is fundamental to theory and practice. First, there is a lack of understanding regarding when and how to use these approaches to select the specific level (degree) of the innovation project (from incremental to more radical ones). Likewise, without a proper method, firms might not be able to select the opportunities related to radical innovations (Brasil, Salerno, & Gomes, 2018). Using a kind of method that fits all approaches, firms might fail in differentiating the valuation and selection of incremental and radical innovation. Finally, it is also important to determine whether newer PPS methods meet the particular features of radical innovation projects in order to provide a guide for future research.

This paper provides important insights that advance the knowledge on PPS literature. First, we show how PPS evolved. This is especially important to new scholars in the field, who might face difficulties in building a big picture of how this field is structured. Second, we identified the main characteristics of different PPS approaches. This finding provides a rich portrait regarding the approaches, helping scholars and practitioners to identify when and how to use these approaches according to the innovation level. Finally, we provide a theoretical framework in which we associate these approaches according to the innovation level.

This study is structured in five sections as follows. Section 2 presents the methodological procedures including the documents sample selection and the analysis of the methods. Section 3 discusses the PPS approaches, describing each of the 12 taxonomies identified. In Section 4, we discuss the key issues for radical innovation portfolio management and which approaches are more suitable to this end. In addition, we elaborated an integrated portfolio selection method for radical innovation framework, based on the main characteristics and advantages of the approaches. Finally, Section 5 is concluded with the final considerations and future research directions.

2. Methodology

To obtain an overview of the evolution of PPS, a systematic literature review was performed. This methodology enforces explicit methods to perform a thorough research and analysis, underlining how a determined field has evolved, and enabling the identification of consensus or divergence among studies and theories (Briner & Denyer, 2010).

2.1 Sample selection procedure

The collecting methods’ process started through the search on the Web of Science database, which provides different searching and browsing options to obtain higher accuracy during the research. We applied the keywords: “Project Portfolio Selection,” in order to gather the broadest number of related works. The search returned 175 documents; this sample was refined selecting only the pertinent predefined categories as follows: “management,” “operations research management,” “engineering industrial” and “business.” Additionally, we focused our research only on “article,” because this type of documents is peer-reviewed – an important indicator of quality. We then reached out 75 documents.

Another selecting parameter was the inclusion of articles whose purpose was the proposition or improvement of method or approach for PPS. We read again the titles, abstracts and keywords. Based on this process, 35 articles related to “Project Portfolio Selection” were obtained. Notwithstanding, four documents referenced by Cooper, Edgett, and Kleinschmidt (1999) were added and other works were included too because they are recurrently cited by the studies in our sample. In total, 48 articles were selected.

2.2 Methods analysis procedure

Once the sample of articles was established, the first stage was to read them identifying the relevant points and the main contributions including proposed frameworks, techniques, tools, examples of application and required data. Based on Cooper et al.’s (1999) portfolio selection methods classification, we defined 12 taxonomy categories to classify the PPS approaches. The categories are: financial, probabilistic, options pricing, strategic, scoring, combinatorial optimization, behavioral, mapping approaches, real options, integrated methods, information gap theory and scenario-based approach. Table I shows the number of documents per year and methods categories. Combinatorial optimization (15) and integrated methods (13) have the greatest number of approaches and also represent the categories more explored in the last 10 years (ten and five documents, respectively).

Afterward, we carefully examined each PPS approach. We then identified four key requirements that PPS methods must meet in order to address radical innovation projects: dynamism, interdependencies, uncertainties and required input data. Based on these analyses, we proposed an integrated portfolio selection method for radical innovation (Section 5).

3. Project portfolio selection methods

According to Cooper et al. (1999), portfolio selection methods have progressively changed since the first methods emerged in 1960. At that time, methods were strongly focused on mathematical techniques. Over time, because the business and technology environment became more complex, more sophisticated methods emerged. Scholars identified that some of these approaches consist of prescribed solutions (one fits all), which might not fit with the vast variety of project and firms’ conditions (Chien, 2002; Baker, 1974). Furthermore, these prescribed aggregate methods raise a paradigm wherein a given set of projects (usually competing for limited resources) must be appraised in order to choose the ones which will maximize profits. Although these methods might be useful for more incremental innovation, an increasing research stream suggested that prescribed methods are not sufficient to analyze radical innovation projects due to the large numbers of unpredictable variables and knowledge gaps (e.g. Baker, 1974; Archer & Ghasemzadeh, 1999; Ghapanchi, Tavana, Khakbaz, & Low, 2012). In the following sections, we present 12 methods for PPS.

3.1 Financial methods

In these methods, the most important selection criterion is profitability and the objective is to establish a portfolio that maximizes expected returns (Bard et al., 1988). This is one of the most widely adopted methods by managers for screening projects (Chien, 2002; Dutra, Ribeiro, & Carvalho, 2014), and basically, are utilized to assess projects individually. Two documents were identified within this category (Table II).

3.2 Probabilistic methods

Among the most popular probabilistic methods are Monte Carlo simulation and Bayesian theory. These methods analyze different scenarios producing distributions of possible outcome values. Some methods are stochastic, using random values. Archer and Ghasemzadeh (1999) indicate that simulation techniques are more effective to analyze risk (Table III).

3.3 Option pricing theory

Option pricing uses mathematical models to calculate how demand varies at different prices’ levels. These models work with three critical pricing elements: pricing strategy, both buyer and seller value of the product, and techniques and elements that affect profitability (Galai & Masulis, 1976). According to Chien (2002), mathematical models are slightly used by managers because these models tend to do not consider the diversity of projects and criteria (Table IV).

3.4 Strategic methods

Since selecting the right projects is extremely critical, not just to maintain the competitive position, but also to achieve financial goals (Cooper et al., 2002), authors have been exploring selection methods driven basically by strategic objectives. Zeynalzadeh and Ghajari (2011) raise particular awareness about the hierarchy of strategy (i.e. organization, portfolio and project strategy) to be considered during the process of project selection. Strategic Buckets is one of the most widespread methods, in which managers create budget buckets where projects are categorized and prioritized (Cooper et al., 2002). These methods involve both external and internal considerations of the firm (Archer & Ghasemzadeh, 1999), enabling a focused perspective from the entire context (Table V).

3.5 Scoring methods

Basically, projects are screened based on a set of different requirements and aspects. Projects are eliminated if they do not meet the requirements and selected if they meet most part of these requirements (Archer & Ghasemzadeh, 1999). Scoring methods are a quantified extension of a checklist and rely on subjective ratings of managers. This fact presents a weakness because such ratings and weights are arbitrarily determined (Cooper, 1981), and this is one of the reasons why scoring methods could work better with objective-centered approaches. This method is also popular in practice because of its simplicity of execution (Hall & Nauda, 1990) (Table VI).

3.6 Combinatorial optimization

This category includes methods such as multi-criteria decision-making methods (MCDM), data envelopment analysis and analytical hierarchy approaches (AHP). According to Razi and Shariat (2017), MCDM are based on the utility theory and assist the human constraints to deal with profits and loss. In AHP models, criteria and alternatives are arranged in hierarchical structures in at least three levels, in order to prioritize the criteria within each level (Amiri, 2010).

According to Archer and Ghasemzadeh (1999), these methods are complex and difficult to apply, and also require a lot of information. Another constraint stems from the use of Technique for Order of Preference by Similarity to Ideal Solution and crisp values which are not recommended for handling uncertainties since these techniques work with single numeric values (Amiri, 2010). However, robust and specialized assessing algorithms may be useful for early screening when there is a large number of potential projects (Beaujon et al., 2001; Abbassi, Ashrafi, & Tashnizi, 2013). The approach proposed by Amiri (2010) incorporates fuzzy numbers, enabling managers to introduce interval judgments improving handling uncertainty issues (Table VII).

3.7 Behavioral methods

These methods aim to create a consensus based on experts and managers’ opinions to decide which projects should be undertaken. For Cooper et al. (1999), this taxonomy is particularly useful during the initial stages of the project selection, in which, virtually, these opinions might be the single input data available (Table VIII).

3.8 Mapping approaches

In this taxonomy, projects are plotted against various parameters in form of matrices or bubble diagram in which it is possible to visualize the strategic position of the firm’s product (Archer & Ghasemzadeh, 1999). Mapping methods have gained wide popularity among companies. For instance, Linton, Walsh and Morabito (2002) mentioned firms, such as 3M, Lucent Technologies or Procter and Gamble, which have implemented such mapping method (Table IX).

3.9 Real options

These methods incorporate learning models to resolve uncertainties through the experience gained over time. Differently from traditional economic analysis (e.g. discounted cash flow), the real options model is a fluctuating investment decision and assumes that the strategy may evolve depending on the real characteristics of the environment (Mun, 2005). It also helps managers to define a portfolio (which projects and when they should be run) among several seemingly economic outcomes and projects. Brasil et al. (2018) mention some barriers to adopt real options in PPS by firms: mathematical complexity, non-intuitive outcomes, and difficulties to handle unknown uncertainties, among others (Table X).

3.10 Integrated methods

Integrated methods are those in which two or more different taxonomies are employed in order to guide the portfolio selection process (Table XI).

3.11 Information gap theory

This method assists the decision-making process in situations under several knowledge gaps. It works with a mathematical process model, performance requirement and uncertainty model (Regan et al., 2005). Through the information gap model, analysts can identify what they believe to be true and important regarding the system, enabling the evaluation of robustness (identifying possible bad outcomes) and opportunity (windfall chances).

Differently from traditional selection methods, which are based on maximum expected benefits, in this method, the focus is on the identification of knowledge gaps, uncertainties and risks, in order to determine the least vague portfolio (Table XII).

3.12 Scenario-based approach

One of the main concerns of this approach is to address exogenous uncertainties, which are defined by Liesiö and Salo (2012, p. 162) as missing data or information “which are not influenced by the projects” (e.g. macroeconomic developments or rate of industry growth). This approach proposes to deal with exogenous uncertainties through the scenarios technique. Scenario planning enables a company to imagine possible futures through exploring “the joint impact of various uncertainties” (Shoemaker, 1995, p. 26). For Shoemaker (1995, p. 27), scenarios “attempt to capture the richness and range of possibilities, stimulating decision makers to consider changes they would otherwise ignore.”

The methods rely on the assumption that, through mathematical models and functions, it is possible to complete the information provided by experts, to build a scenario and identify synergies and probabilities. Scenario-based methods require specific and sophisticated knowledge to operate, thus, its applicability and diffusion are restricted (Archer & Ghasemzadeh, 1999) (Table XIII).

4. Results and discussion

4.1 Portfolio selection for radical innovation projects

In this section, we develop a discussion by bridging incremental and radical innovation levels and PPS approaches. First, integrated methods (i.e. those that enable managers to employ different techniques and different input data) are more convenient to adopt in both cases – incremental and radical innovations (Archer & Ghasemzadeh, 1999; Cooper et al., 1999; Dutra et al., 2014). This is also especially relevant for radical innovation, since no single method has all the features required to respond adequately under uncertain circumstances. Coldrick, Longhurst, Ivey, and Hannis (2005) recognize that optimization methods are powerful, but if combined with scoring, behavioral or mapping methods, they would be more beneficial. Second, qualitative judgment is more appropriate to deal with radical innovation (Archer & Ghasemzadeh, 1999; Cooper, 1981; Fahrni & Spätig, 1990). For instance, mapping or scoring methods are very useful to identify which projects to cut-off regarding the number of uncertainties and risks. Quantitative methods seem to be more suitable for handling risk (Fahrni & Spätig, 1990), rather than uncertainty in which the probabilities are “unknowable” (Brasil et al., 2018). However, some techniques employed by quantitative methods might handle uncertainties, such as is the case of fuzzy theory (combinatorial optimization) or volatility metrics (real options) (Jafarzadeh, Akbari, & Abedin, 2018; Wang & Hwang, 2007).

Another important aspect to consider when handling uncertain projects is the method’s ability to update knowledge, enabling managers to learn through the experience gained. Real options and probabilistic methods that implement Bayesian network enable the aggregation of information at each stage of the process (Chin, Tang, Yang, Wong, & Wang, 2009). Finally, an interesting technique of trade-off between immunity to uncertainty and aspirations is the information gap theory model (Regan et al., 2005). This method focuses on the identification of knowledge gaps, uncertainties and risks in order to determine the tolerance for undesirable outcomes.

Based on the literature review and the considerations presented above, we identified four key requirements that PPS methods must meet in order to address radical innovation projects. First, Dynamism, is defined as the characteristic that enables managers to permanently adjust the portfolio according to the evolution of the firm’s guidelines (Kavadias & Chao, 2008, p. 139). Second, Interdependencies, is regarded as the interrelations among projects and their impacts on portfolio performance (i.e. synergy or incompatibilities); several authors suggest that the identification of interdependencies is crucial for innovation projects (Verma & Sinha, 2002; Chien, 2002; Archer & Ghasemzadeh, 1999). Third, Uncertainties, is perceived as situations where neither the outcomes nor the odds could be known (Sommer, Loch, & Pich, 2008). The concept of uncertainty is central to radical innovation. Leifer, O’Connor, and Rice (2001) explain that radical innovation processes involve unprecedented situations and features plenty of technical and market uncertainties (p. 103). Finally, Required input data that refer to the kind of data required by the PPS method. Some authors (e.g. Baker, 1974; Fahrni & Spätig, 1990) consider that financial input data (e.g. estimated costs, profit margin, investment, sales, etc.) imprecise because information will never be precisely available. For Ghasemzadeh and Archer (2000), qualitative information is more recommended for assessing radical innovation projects.

Regarding these four key issues and based on the information provided in the studies, we built a framework (Figure 1). To do so, the approaches identified were analyzed by considering the adequacy between the approaches and the four requirements – we developed a scoring model in which each approach received a point or more if such approach is suitable to one characteristic or more (see Table AI). According to this assessment, behavioral methods and information gap theory are the two approaches more suited for radical innovation, followed by real options and integrated methods. The methods less appropriate for radical innovation are financial methods, options-pricing theory and scoring methods; these approaches are more suitable for incremental innovation (Fahrni & Spätig, 1990).

4.2 Integrated portfolio selection method for radical innovation framework

After identifying the advantages and disadvantages of each approach, and identifying the methods more recommended to radical innovation projects, we developed a framework for supporting the PPS selection method according to the innovation level. This framework might help managers to choose and to combine different methods among the alternatives according to a specific innovation level (i.e. radical or incremental innovation).

The framework comprises three phases: ideation phase, which includes the strategic goals, resource constraints and project guidelines; the phase ends with the analysis of the characteristics of the project in which the innovation level is identified. The second phase is the PPS process, where the projects are screened using one or more approaches according to the innovation level. In the last phase, decision makers define the project portfolio based on the recommendations of the selection approaches; this phase also comprises the portfolio adjustments stage, in which managers might include, modify or discard projects in accordance with the environment and the strategic goals. Figure 2 presents the framework suggested.

5. Conclusions and future research

In this paper, we addressed the question of how the selection approaches have evolved to better fit within radical innovation conditions by conducting a systematic literature review. Based on this research approach, we showed how the project selection methods evolved. We present a comprehensive characterization of these methods. We have also identified four key aspects to manage radical innovation portfolios: dynamism, interdependency management, uncertainty treatment and required input data. Based on these findings, we propose a framework for supporting the PPS process according to the innovation level. Moreover, we suggest which PPS approach is more appropriate to incremental and radical innovation projects.

Of course, there is an important opportunity for further studies. We have not discussed in details about how to test our framework. Further studies might test our framework in different industry settings. Also, scholars might test if the methods related to incremental and radical innovation provide superior performance (in relation to the configuration of the portfolio). Scholars might test the best set of methods according to the project’s features (incremental and radical innovation). Finally, scholars might explore empirically how different combinations of methods are more recommended taking into account different industry settings and other features.

Figures

Figure 1

Portfolio selection methods: from incremental to radical innovation

Figure 2

Integrated portfolio selection method for radical innovation framework

Table I

Number of documents per year and methods categories

	Total	1974	1976	1981	1985	1988	1990	1992	1999	2000	2001	2002	2003	2004	2005	2007	2008	2009	2010	2011	2012	2013	2014	2015	2016	2017	2018
Financial methods	2	1				1
Probabilistic models	1																	1
Options-pricing theory	1		1
Strategic models	4						1					1								1				1
Scoring models	3			1			1		1
Comb. optimization	15				1						1	1		1	1		1		1		1	2		1		2	2
Behavioral models	1										1
Mapping approaches	2							1					1							1
Real options	3															1										1
Integrated models	13								1	2	1	2			1	1							1	1	1	2
Info. gap theory	1														1
Scenario-based approach	2																				1				1
Total	48	1	1	1	1	1	2	1	2	2	3	4	1	1	3	2	1	1	1	2	2	2	1	3	2	5	2

Table II

Methods’ analysis: financial methods

Author (year)	Summary	Techniques	Projects	Data required
Bard, Balachandra, and Kaufmann (1988)	Projects are assessed in two stages: screened against critical factors and with the other projects. This is achieved through mathematical programming technique that maximizes expected returns	Nonlinear integer programming NPV. Zero-one decision Decision support system (DSS)	Small computer firm specializing in peripheral equipment	Exogenous critical factors (regulations) Project-related variables Financial Data
Lal (1974)	Projects are selected according to the cost and benefits and several financial portfolio selection techniques	IRR Marginal social cost Marginal social value NPV Value of marginal product Pareto optimality	Food industry Trade companies Several industrial firms	Expected benefits Financial data Production and external information (trade)

Table III

Methods’ analysis: probabilistic methods

Author (year)	Summary	Techniques	Projects	Data required
Chin, Tang, Yang, Wong, and Wang (2009)	This methodology assesses new product development risk by Bayesian network with a systematic probability generation and using an NPD project risk network and uncertain variables	Directed acyclic graph (DAG) Conditional probability table (CPT) Bayesian network NPD project risk network Pair-wise comparison matrix	Multinational flashlight manufacturer	Project-related variables Available resources Conditional probabilities Expected benefits Financial data Strategic goals

Table IV

Methods’ analysis: option pricing theory

Author (year)	Summary	Techniques	Projects	Data required
Galai and Masulis (1976)	This method combines the option pricing model (OPM) with the capital asset pricing model (CAPM) to decide about future investments	Black-scholes Capital asset pricing model CAPM Linear function Economic return (NPV, IRR, ROI, RAI, PBP and EV) Cash flow models European-type options	Two private sector companies	Expected benefits Financial data Production and external information (trade)

Table V

Methods’ analysis: strategic methods

Author (year)	Summary	Techniques	Projects	Data required
Cooper et al. (2002)	Managers define different resources buckets where projects are sorted and prioritized according to these goals: value maximization, balance (long vs short time, or high vs low risk), strategic direction and the right number of projects	Strategic buckets Familiarity matrix	Several large industrial firms	Strategic goals Product lines Project type Scoring criteria Expected benefits Financial data
Fahrni and Spätig (1990)	The guidelines are concentrated on the most critical problems, the degree of quantification of relevant factors, interdependencies, consideration of single or multiple objectives and degree of risk and uncertainty	Decision trees Checklists Analytical hierarchy process (AHP) Economic return (NPV, IRR and ROI) Simulation Linear and integer optimization Mathematical programming	Several Swiss firms	Project-related variables Interdependencies Expected benefits Scoring criteria
Zeynalzadeh and Ghajari (2011)	The framework considers the firm’s strategy, single project success, portfolio balance and interdependencies. The diamond technique is used to analyze the novelty, technology, complexity and pace	Diamond technique Scoring system	Not informed	Project-related variables Interdependencies Strategic goals Financial data Project success criteria Project competitive advantage
Jafarzadeh, Tareghian, Rahbarnia, and Ghanbari (2015)	This method uses reinvesting strategy within a flexible time horizon. In order to maximize gains, the model aims to optimize the combination of projects, time horizon and schedule of projects	Linear and integer optimization Functions and algorithms Nonlinear integer programming	Illustrative example	Project-related and decision variables Strategic goals Time horizon Interdependencies

Table VI

Methods’ analysis: scoring methods

Author (year)	Summary	Techniques	Projects	Data required
Cooper (1981)	This method considers the compatibility of resources, the novelty of the project, market competitiveness, technical complexity, competitive strength and economic advantage of the product. It also includes multiple regression analysis to determine the degree of success or failure	Checklists Multiple regression analysis Cross-split-half method Test of predictability	Several large industrial firms	Project-related variables Expected benefits Strategic goals Available resources
Hall and Nauda (1990)	An interactive process for R&D selection based on the judgment of key business and technical elements: this approach emphasizes both corporate strategic planning and customer needs	Peer-review/Expert scoring process Experts/Market/Customer Surveys	Department of Defense Aerospace Contractor	Project-related variables Expected benefits Available resources Comparison criteria Strategic goals
Henriksen and Traynor (1999)	PPS method based on the ranking of relevance, risk, reasonableness and return. This technique implements an index algorithm which produces a measure of project value as a function of both merit and cost	Scoring algorithm Peer-review scoring process Stakeholder inquiry Delphi technique Linear mapping Value index equation Decision support system (DSS)	Federal Research Laboratory	Project-related variables Expected benefits Comparison criteria Strategic goals

Table VII

Methods’ analysis: combinatorial optimization

Author (year)	Summary	Techniques	Projects	Data required
Amiri (2010)	This method uses six criteria to compare investment alternatives through an AHP and fuzzy TOPSIS techniques	Analytical hierarchy process (AHP) Fuzzy TOPSIS Portfolio matrices Delphi technique	National Iranian Oil Company	Project-related variables Expected benefits Strategic goals Financial data
Beaujon, Marin and McDonald (2001)	The mathematical model derives from a multidimensional knapsack problem and applies specialized algorithms	Mixed-integer programming (MIP) Linear programming (LP) Manual solution Heuristics Sensitivity Analysis	General Motors R&D Center	Strategic goals Available resources Project-related variables Expected benefits Financial data
Doerner, Gutjahr, Hartl, Strauss, and Stummer (2004)	The Ant Colony approach imitates the ants’ behavior to create the shortest and most efficient paths. This method enables the identification of the best configuration for the project portfolio	Multi-objective combinatorial optimization Pareto Ant Colony Optimization Algorithm Quadtree data structure Pareto Simulated Annealing Non-Dominated Sorting Genetic Algorithm (NSGA)	Illustrative example	Project-related variables Interdependencies Expected benefits Financial data Strategic goals Available Resources Synergies among projects
Loch and Kavadias (2002)	The method uses marginal analysis to optimally allocate the scarce budget. It considers multiple interacting factors: interdependencies, common resource pool, increasing or decreasing returns, carry-over of the investment benefit over multiple periods, uncertainty and risks	Risk-return matrices Market segment balances Functions and algorithms Weighted average cost of capital (WACC)	Illustrative example based on the diamond industry	Project-related variables Available Resources Expected benefits Financial data Interdependencies Synergies among projects
Mandakovic and Souder (1985)	A hierarchical zero-one model of the R&D budgeting process. The authors suggest a flexible or “in-house” objective function that leaves an open option for the users	Multiple-criteria objective functions Economic Return (NPV, IRR, RA, RR and EV) Ad hoc objective function Decomposable interactive algorithm Hierarchical framework Random number generator	Illustrative example	Project-related variables Risk coefficient Expected benefits Probability of success Financial data Available Resources Weighted contribution per project
Mohanty, Agarwal, Choudhury, and Tiwari (2005)	The proposed approach simultaneously addresses the issue of combining both monetary and non-monetary aspects using fuzzy set theory to overcome the vagueness in the preferences	Pair-wise comparison matrix Triangular fuzzy numbers Analytical network process (ANP) Fuzzy cost analysis Fuzzy set theory Market/customer surveys Feasibility studies Fuzzy ANP framework Cash flow models Supermatrix	Iron and steel industry in South East Asia	Project-related variables Interdependencies Expected benefits Financial data Strategic goals Available Resources Past projects information Exogenous critical factors
Abbassi, Ashrafi, and Tashnizi (2013)	The authors proposed a 0-1 nonlinear mathematical programming method for balancing portfolio following the Markowitz portfolio logic	Cross-entropy algorithm Functions and algorithms Nonlinear integer programming	Illustrative example	Interdependencies Strategic goals Project-related variables Resource constraints Financial data Exogenous critical factors Technological plan
Razi and Shariat (2017)	This method consists of hybrid artificial neural network algorithms based on the gray relational analysis (GRA), decision tree and regressions	Grey Relational Algorithm C&R tree (CART) VIKOR Fuzzy TOPSIS Decision matrix Expert opinion questionnaire Sensitivity analysis	Iran Power Plant Repair Company	Strategic goals Project-related variables Evaluation criteria and weighting
Ghapanchi, Tavana, Khakbaz, and Low (2012)	The model is a mathematical programming approach used to compare the inputs and outputs of decision-making units (DMU)	Data envelopment analysis (DEA) Fuzzy DEA (FDEA)	Illustrative example: information system/information technology (IS/IT) projects	Interdependencies Strategic goals Evaluation criteria and weighting Project costs Contribution margins Fuzzy risk
Pérez, Gómez, Caballero, and Liern (2018)	The method enables the identification of synergies, incompatibilities and other constraints through mathematical programs. It uses fuzzy parameters and adopts several graphics to communicate the results	Fuzzy parameters Mathematical programming	Illustrative example and a real case in a Spanish state university	Project-related variables Financial data Strategic goals Political constraints Available resources Past projects information Time horizon
Liesiö, Mild, and Salo (2008)	This method supports PPS through multiple evaluation criteria, advanced portfolio-level benefit-cost analysis and interdependencies identification	Multi-objective zero-one linear programming Interval-valued objective function coefficients Dynamic programming algorithms Portfolio-level benefit-cost analysis Multi-objective zero/one linear programming (MOZOLP)	Illustrative example: technology company	Evaluation criteria and weighting Project-related variables Financial data Strategic goals Resource constraints
Hall, Zhuoyu, Qi, and Sim (2015)	This approach uses riskiness indexes, integrates correlations and synergies. The selection is centered on target based choice instead of maximizing returns	Underperformance riskiness index (URI) Entropic underperformance riskiness index (EURI) Benders decomposition techniques Functions and algorithms Sensitivity analysis	Illustrative example (200 projects)	Projects-related variables Multiple objectives and constraints Uncertainties Interdependencies
Tavana, Khalili-Damghani, and Abtahi (2013)	This method is a hybrid fuzzy multidimensional multiple-choice Knap-sack problem (MMKP) that helps to define which project is the most indicated under certain circumstances of limited resources	Efficient Epsilon-Constrain (EEC) method Multi-objective evolutionary algorithm Data envelopment analysis (DEA) Fuzzy set theory	Case study: local bank (East Virginia, EEUU.)	Resource constraints Projects-related variables Strategic goals Past projects information
Jafarzadeh, Akbari, and Abedin (2018)	Aiming to address simultaneously prioritization, uncertainty and interdependencies, the authors develop a model based on three stages: modeling the problem, house of quality (HOQ) and evaluation of maximal portfolio	Data envelopment analysis (DEA) Fuzzy set theory Quality function development (QFD) Crisp values HOQ matrix	Numerical example of a large-scale Telecommunication Company	Projects-related variables Strategic goals Customer requirements Interdependencies Risk and dependencies among projects Financial data
Kalashnikov, Benita, López-Ramos, and Hernández-Luna (2017)	This optimization method considers interdependencies among projects, resource limitations and addresses the problem as a Mixed-integer quadratically constrained programming problem	Lean Six Sigma 0-1 bi-objective quadratic programming problem CPLEX Functions and algorithms Mixed-integer quadratically constrained programming (MIQCP)	Illustrative example	Number of potential projects and related variables Strategic goals Resource constraints Financial data Interdependencies Time required by projects

Table VIII

Methods’ analysis: behavioral methods

Author (year)	Summary	Techniques	Projects	Data required
Han and Diekmann (2001)	Based on the CIA (cross-impact analysis) technique and critical conditions, the interactions among the projects are predicted in order to identify uncertainties and data gaps, enabling decision makers to test combinations to obtain the best project portfolio	Cross-impact analysis Go/No-go decision analysis model Decision makers group	International government projects	Critical conditions Project-related variables Rate impact of the decision strategies Interdependencies

Table IX

Methods’ analysis: mapping approaches

Author (year)	Summary	Techniques	Projects	Data required
Freedman and Desi (2011)	The DICE framework assesses the risks and probabilities of success of projects. It is based on these factors: duration of the project, the integrity of the team, commitment to change and the effort of stakeholders. This chart distributes the projects into three areas: comfort, risk and uncertainty	BCG’s DICE framework BCG analysis Statistical trend analysis	Power plant	Project-related variables Scoring criteria Expected benefits Financial data Customer data Technical data Strategic goals
Wheelwright and Clark (2003)	This method helps to determine which new projects must be adopted and when, how the project set evolves over time and what role each project should play in the overall development effort. This method is useful to identify the gaps, to balance the portfolio and to reveal where development capabilities should be strengthened	Aggregate project plan Bubble diagram Project sequence chart	PreQuip, scientific instruments company	Project-related variables Project types Expected benefits Strategic goals Resources required per project

Table X

Methods’ analysis: real options

Author (year)	Summary	Techniques	Projects	Data required
Luehrman (1998)	In this method, the author explores how option pricing can be used to improve decision making about the sequence and timing of a portfolio of strategic investments. Through the option space, managers can select projects according to the value metrics (value-to-cost and volatility)	Option pricing Option space value-to-cost metric (NPVq) Volatility metric Cash flow models Nested call options	Illustrative example. WeatherIze Corporation (fabrics)	Project-related variables Available resources Expected benefits Financial data Interdependencies Planning periods
Wang and Hwang (2007)	A fuzzy set approach for R&D portfolio selection by using real options valuation model. The objective of this method is to maximize the total benefits of the R&D investment portfolio	Fuzzy set theory Zero-one and probabilistic integer programming Crisp mathematical model Option pricing Fuzzy compound option Triangular fuzzy numbers Qualitative possibility theory Geske compound option Volatility metric Newton-Raphson algorithm	An illustrative example from the pharmaceutical industry	Project-related variables Available resources Expected benefits Planning periods Strategic goals
Montajabiha, Khamseh, and Afshar-Nadjafi (2017)	The PPS approach applies n-fold compound real options valuations giving flexibility to managers to decide which projects to select. In this method, uncertainty is treated as a range, eliminating the need for statistical distributions	Robust combinatorial optimization algorithm N-fold compound option model Sensitivity analysis Functions and algorithms Robust mixed-integer programming (MIP)	An illustrative example in the pharmaceutical industry	Project-related variables Budget restrictions Estimated financial data Past projects information Strategic goals

Table XI

Methods’ analysis: integrated methods

Author (year)	Summary	Techniques	Projects	Data required	Integrated taxonomies
Archer and Ghasemzadeh (1999)	A staged and integrated framework where users are able to select the most suitable techniques for each stage. The approach is divided into three main phases: strategic considerations, individual project evaluation, and portfolio selection. It also has a stage of portfolio adjustment whereby users can make some changes	Economic return (NPV, IRR and ROI) CAPM (capital asset pricing model) Work breakdown structure (WBS) Monte Carlo simulation Bayesian statistical theory Market research Ad hoc approaches based on scoring Q-sort Portfolio matrices	Experimental tests	Project-related variables Scoring criteria Expected Benefits Strategic goals Financial Data Interdependencies	Strategic Financial Scoring Mapping Combinatorial optimization Probabilistic Behavioral
Chien (2002)	The approach has three phases: identification of objectives of the portfolio, construction of scales to measure portfolio attributes, and measurement of alternative portfolios. The model focuses on the identification and exploitation of the interrelations among projects	Work breakdown structure (WBS) Value/utility functions MAV/MAU models Linear programming (LP)	An illustrative example based on the project: American stop smoking intervention study by The National Cancer Institute	Project-related variables Scoring criteria Expected Benefits Financial data Strategic goals Interdependencies	Scoring Combinatorial optimization Behavioral
Coldrick, Longhurst, Ivey, and Hannis (2005)	This staged method is an extension of Lockwood’s (1999) work in which the selection process is divided into project evaluation (single projects), project selection (examining a range of projects), and application to the different types of research	Checklists Delphi technique Scoring table layout Risk/reward assessment Bubble diagram Risk analysis Economic return (NPV, IRR, CBA, DCF and PVI)	Gas turbine project	Project-related variables Scoring criteria Expected Benefits Strategic goals Financial data	Combinatorial optimization Scoring Behavioral Mapping Financial
Cooper, Edgett, and Kleinschmidt (2000)	In this evolved version of the Stage Gate System®, the two-parts decision gates (Go/Kill) were incorporated, and also a formal stage where projects are force-ranked against one another	Strategy buckets approach Bubble diagrams Prioritized scored list Economic return (NPV, IRR and EV) Checklists	Several large industrial firms	Project-related variables Scoring criteria Expected benefits Financial data Strategic goals	Strategic Financial Scoring Mapping
Dutra, Ribeiro, and Carvalho (2014)	An economic-probabilistic model that combines financial and probabilistic approaches to consider uncertainties, incorporate multiple criteria, and to adjust the method in response to changes in the business environment	Economic return (NPV, IRR, RA, RR and EV) PayBack period Monte Carlo Decision Support System (DSS) Triangular distribution	Power distribution company	Project-related variables Scoring criteria Expected benefits Financial data Strategic goals	Financial Probabilistic
Ghasemzadeh and Archer (2000)	An evolution of the integrated framework for project portfolio selection (1999). The authors set up the project analysis and selection system (PASS), an organized framework for portfolio selection through a decision support system (DSS)	Economic return (NPV, IRR, ROI and RAI) Analytical hierarchy process (AHP) Decision support system (DSS) Group support system (GSS) Expert choice Integer linear programming Ad hoc approaches based on scoring Gantt Chart Portfolio matrices	Experimental tests with two high-tech companies	Project-related variables Scoring criteria Interdependencies Expected benefits Strategic goals Financial data	Strategic Financial Scoring Mapping Combinatorial optimization Probabilistic Behavioral
Linton, Walsh, and Morabito (2002)	This dual method uses a multi-criteria decision-making model and a graphical decision support system that enables the integration of objective and subjective methods for analysis and selection. This approach is ideal for reducing a large number of projects	Data envelopment analysis (DEA) Value creation model (VCM) Parabox representation Nicheworks depiction	Advanced technology division of bell laboratories	Project-related variables Stage of the intellectual property lifecycle Expected benefits Financial data Strategic goals	Scoring Mapping Combinatorial optimization
Loch, Pich, Terwiesch, and Urbschat (2001)	This methodology comprises the identification of target contributions, project expectations, and needs. Projects are selected by mixed-integer linear programming. This approach focuses on qualitative data such as the criteria, project contributions and gaps	Matrix of evaluation criteria Mixed-integer programming (MIP) Linear programming (LP) Checklists	R&D at BMW	Project-related variables Interdependencies Scoring criteria Expected benefits Financial data Strategic goals	Scoring Mapping Combinatorial optimization
Medaglia, Graves, and Ringuest (2007)	An evolutionary method which includes multiple objectives, project interdependencies and linear structure for resource constraints. The method is ideal for partially funded projects, multiple objectives, uncertainties, project interdependencies, and linearly-constrained resources	Parameter space investigation method (PSI) Black Box approach Pareto solution Multi-objective evolutionary algorithms (MOEA) Monte Carlo simulation	Experimental tests	Project-related variables Scoring criteria Expected benefits Financial data Interdependencies	Combinatorial optimization Probabilistic
Jeng and Huang (2015)	The method consists of an integrated approach based on a hybrid MCDM, modified Delphi, a DEMATEL method and an analytic network process. The objective is to overcome the complexity of interdependences and feedbacks between criteria and alternatives	Modified Delphi method Need solution differentiation and benefits (NSDB) framework DEMATEL method ANP	National research institute	Strategic goals Evaluation criteria Potential customer requirements	Combinatorial optimization Strategic Behavioral
Xidonas, Doukas, Mavrotas, and Pechak (2016)	This approach incorporates environmental criteria and uses a bi-objective programming model. In order to handle uncertainties, the model uses the iterative trichotomic approach (ITA) combined with multi-objective problems and Monte Carlo simulations	Multi-objective integer programming (MOIP) Pareto solution Monte Carlo simulation Functions and algorithms Stochastic multi-objective acceptability analysis (SMAA)	40 large firms	Financial data Projects-related variables Multiple objectives and constraints Conditional probabilities Expected Benefits Uncertainties	Combinatorial optimization Probabilistic
Ei-Kholany and Abdelsalam (2017)	This method combines a multi-objective binary cuckoo search (MOCS) algorithm with Monte Carlo simulation to support the PPS decision. Uncertainties are handled via Monte Carlo simulation to translate risk into quantitative data	Monte Carlo simulation MOCS Fuzzy parameters Mathematical programming Functions and algorithms Meta-heuristic algorithm Binary cuckoo search Pareto solution	Illustrative example (40 projects)	Criteria and performance values Project-related variables Multiple-objective parameters	Combinatorial optimization Probabilistic
Hu and Szmerekovsky (2017)	On the basis of limited resources and the Newsvendor problem perspective, the authors develop a set of procedures and heuristics to provide managerial guidelines in the portfolio selection process. The aim of the model is to find the ideal projects, avoiding cost overruns and budgetary slack	Functions and algorithms Sensitivity analysis Evolutionary solution method (Excel solver) Heuristics based on projects ratio and profit ratio	Numerical example	Number of potential projects and related variables Strategic goals Resource constraints Financial data Budget by project	Combinatorial optimization Strategic

Table XII

Methods’ analysis: information gap theory

Author (year)	Summary	Techniques	Projects	Data required
Regan, Ben-Haim, Wilson, Lundberg, Andelman, and Burgman (2005)	This method uses information gap theory to propagate uncertainties, by establishing a tolerance threshold to rank management options and to cut off pernicious options. This approach includes three elements: a mathematical process model, a performance requirement and a model for uncertainty	Robustness Function Decision table (utilities and probabilities) EV Robustness curves	Decision problem for conservation area of the Sumatran Rhino	Project-related variables Positive and negative expected outcomes

Table XIII

Methods’ analysis: scenario-based approach

Author (year)	Summary	Techniques	Projects	Data required
Liesiö and Salo (2012)	This method offers interactive decision support for rejection and selection of projects, which uses scenarios probabilities and utility functions. The method uses MOZOLP to identify the non-dominated portfolios which satisfy criteria such as resource and feasibility constraints	Multi-objective zero/one linear programming (MOZOLP) Functions and algorithms Experts opinion survey Risk utility function Tornado plots Conditional value at risk (CVaR)	An illustrative example at the International Institute for Applied Systems Analysis (IIASA)	Project-related variables Experts’ estimates (risk preferences/feasible scenarios) Time horizon
Zhang, Yang, Dou, and Jiang (2016)	The authors proposed a scenario-based approach to select army engineering and manufacturing development projects under incomplete scenario information and interdependency constraints. The method is based on a multi-objective programming model	Scenario-based programming Functions and algorithms Pair-wise comparison matrix Hidden Markov models (HMMs)	An illustrative example of army engineering and manufacturing development project	Experts’ estimates (risk preferences/feasible scenarios) Financial data Project-related variables

Table AI

Detailed values of the innovation adequacy assessment

Document	Method	Dynamism	Interdependencies	Input data	Uncertainty	Total	Mean per method
Bard et al. (1988)	Financial methods	0	0	0	0	0	0.0
Lal (1974)	Financial methods	0	0	0	0	0
Chin et al. (2009)	Probabilistic Models	0	0	0	1	1	1.0
Galai and Masulis (1976)	Opt. pricing theory	0	0	0	0	0	0.0
Cooper and Kleinschmidt (2002)	Strategic models	1	0	0	0	1	2.0
Fahrni and Spätig (1990)	Strategic models	1	1	0	0	2
Zeynalzadeh and Ghajari (2011)	Strategic models	1	1	0	0	2
Jafarzadeh et al. (2015)	Strategic models	1	1	1	0	3
Cooper (1981)	Scoring models	0	0	0	0	0	0.0
Hall and Nauda (1990)	Scoring models	0	0	0	0	0
Henriksen and Traynor (1999)	Scoring models	0	0	0	0	0
Amiri (2010)	Combinatorial opt.	0	0	0	0	0	0.9
Beaujon et al. (2001)	Combinatorial opt.	0	0	0	0	0
Doerner et al. (2004)	Combinatorial opt.	0	1	0	1	2
Loch and Kavadias (2002)	Combinatorial opt.	0	1	0	0	1
Mandakovic and Souder (1985)	Combinatorial opt.	1	0	0	0	1
Mohanty et al. (2005)	Combinatorial opt.	0	0	0	0	0
Abbassi et al. (2013)	Combinatorial opt.	0	1	0	0	1
Razi and Shariat (2017)	Combinatorial opt.	0	0	0	0	0
Ghapanchi et al. (2012)	Combinatorial opt.	0	1	0	0	1
Pérez et al. (2018)	Combinatorial opt.	0	1	0	0	1
Liesiö et al. (2008)	Combinatorial opt.	0	1	0	1	2
Hall et al. (2015)	Combinatorial opt.	0	1	0	0	1
Tavana et al. (2013)	Combinatorial opt.	0	0	0	0	0
Jafarzadeh et al. (2018)	Combinatorial opt.	0	1	0	1	2
Kalashnikov et al. (2017)	Combinatorial opt.	0	1	0	0	1
Han and Diekmann (2001)	Behavioral models	1	1	1	1	4	4.0
Freedman and Desi (2011)	Mapping app.	1	0	0	0	1	2.0
Wheelwright and Clark (2003)	Mapping app.	1	0	1	1	3
Luehrman (1998)	Real options	1	1	1	0	3	2.3
Wang and Hwang (2007)	Real options	1	0	0	1	2
Montajabiha et al. (2017)	Real options	1	0	0	1	2
Archer and Ghasemzadeh (1999)	Integrated models	1	1	1	1	4	2.1
Chien (2002)	Integrated models	0	1	1	1	3
Coldrick et al. (2005)	Integrated models	1	0	0	1	2
Cooper et al. (2000)	Integrated models	0	0	0	0	0
Dutra et al. (2014)	Integrated models	1	0	0	1	2
Ghasemzadeh and Archer (2000)	Integrated models	1	1	1	1	4
Linton et al. (2002)	Integrated models	1	1	0	1	3
Loch et al. (2001)	Integrated models	1	1	0	0	2
Medaglia et al. (2007)	Integrated models	1	1	0	1	3
Jeng and Huang (2015)	Integrated models	1	1	0	0	2
Xidonas et al. (2016)	Integrated models	0	0	0	1	1
Ei-Kholany and Abdelsalam (2017)	Integrated models	0	0	0	1	1
Hu and Szmerekovsky (2017)	Integrated models	0	0	0	0	0
Regan et al. (2005)	Info. gap theory	1	1	1	1	4	4.0
Liesiö and Salo (2012)	Scenario-based	1	0	1	0	2	2.0
Zhang et al. (2016)	Scenario-based	1	1	0	0	2