Meta-choices in ranking knowledge-based organizations

Abstract

Purpose

The purpose of this paper is to address the issue of knowledge visualization and its connection with performance measurement from an epistemological point of view, considering quantification and measurement not just as technical questions but showing their relevant implications on the management decision-making of knowledge-based organizations.

Design/methodology/approach

This study proposes a theoretical contribution that combines two lines of research for identifying the three main meta-choices problems that arise in the multidimensional benchmarking of knowledge-based organizations. The first is the meta-choice problem related to the choice of the algorithm used (Iazzolino et al., 2012; Laise et al., 2015; Daraio, 2017a). The second refers to the choice of the variables to be included in the model (Daraio, 2017a). The third concerns the choice of the data on which the analyses are carried out (Daraio, 2017a).

Findings

The authors show the interplay existing among the three meta-choices in multidimensional benchmarking, considering as key performance indicators intellectual capital, including Human Capital, Structural Capital and Relational Capital, and performances, evaluated in financial and non-financial terms. This study provides an empirical analysis on Italian Universities, comparing the ranking distributions obtained by several efficiency and multi-criteria methods.

Originality/value

This study demonstrates the difficulties of the “implementation problem” in performance measurement, related to the subjectivity of results of the evaluation process when there are many evaluation criteria, and proposes the adoption of the technologies of humility related to the awareness that we can only achieve “satisficing” results.

Citation

Daraio, C., Iazzolino, G., Laise, D., Coniglio, I.M. and Di Leo, S. (2022), "Meta-choices in ranking knowledge-based organizations", Management Decision, Vol. 60 No. 4, pp. 995-1016. https://doi.org/10.1108/MD-01-2021-0069

1. Introduction and contribution of the paper

The article deals with the problem of evaluating business performances in knowledge-based organizations. In particular, we analyze the problem of ranking knowledge-based organizations on performance-related criteria.

This problem is very important when a ranking has to be carried out on a set of different organizations, for example, to select the “best in class.” This problem occurs also in ranking firms to give them awards. Let us think, for example, of the Deming Prize, the Baldridge award, the European Foundation for Quality Management Good Practice Competition and also the problem of evaluating top managers when the company decides to assign them incentives or rewards. Likewise, the problem is important also with reference to knowledge organizations and, in particular, Universities. There are many rankings in this area, such as the ARWU of Shanghai, the QS World University Rankings and the Times Higher Education World University Rankings.

An important element to be underlined is that the issue of measuring organizational performance is a genuine multi-criteria problem. In spite of the traditional way of measuring performance based only on one criterion, that is, the value creation for the shareholder (Jensen, 2002), the performance of the modern enterprise cannot be measured only on one dimension. In the wake of the Balanced Scorecard approach (Kaplan and Norton, 1996) and of Sustainability, it is necessary to adopt a stakeholder's point of view in which many actors have to be considered for correctly evaluating the organizational performance (Iazzolino et al., 2012; Laise et al., 2015).

For knowledge organizations and, in particular, for universities, the capability of existing rankings to measure knowledge creation and provide an effective measure of multidimensional performance has been questioned (Olcay and Bulu, 2017; Vernon et al., 2018) and the main methodological limitations of existing rankings outlined (see e.g. Valmorbida and Ensslin, 2017; Fauzi et al., 2020). Daraio et al. (2015) showed that advanced efficiency analysis techniques can be helpful to overcome the main limitation of existing rankings, that is, their mono-dimensional nature (or the consideration of a few variables), allowing the performance to be evaluated in a multidimensional way.

The theme of strategic decisions and of the methods of knowledge visualization is at the heart of our work. In fact, we will show that in making their decisions, managers are influenced by three main meta-choices, which interact with each other and influence the way in which knowledge is displayed and interpreted. This topic is particularly important for knowledge-based organizations like universities, on which we will carry out an empirical analysis.

Knowledge visualization is an emerging and interdisciplinary research area (Bertschi et al., 2011). There are many different concepts of knowledge visualization in diverse fields. A comprehensive survey of existing concepts is reported in Eppler (2013). Table 1 of Eppler (2013, p. 7) reports the different approaches to knowledge visualization proposed in the literature and cites the epistemological approach to knowledge visualization initiated by Knorr-Cetina (2003) in the second place in terms of number of citations received. The focus of the investigation of Knorr-Cetina (2003) is to try to understand how science creates knowledge and how we know what we know.

We address the issue of knowledge visualization following an epistemological approach. A recent contribution describing the epistemological point of view we are interested in is presented in Carson (2020) and more generally the history of quantification on which Carson relies. As Carson (2020, p. 1) states: “quantification and measurement should be seen not just as technical pursuits, but also as normative ones. Every act of seeing, whether through sight or numbers, is also an act of occlusion, of not-seeing. And every move to make decisions more orderly and rational by translating a question into numerical comparisons is also a move to render irrelevant and often invisible the factors that were not included. The reductions and simplifications quantifications rely on can without question bring great and important clarity, but always at a cost.” In the managerial context, we have the phenomenon of “decisional myopia,” in which the manager decides “to see” only a few dimensions, and perhaps not the important ones, but those more easily available and understandable.

In this work, we propose a theoretical contribution that brings together two lines of research in which the authors have worked in recent years. The first strand concerns the complexity of evaluating the activities of knowledge organizations, which include universities (Daraio, 2017a, b, 2019, 2020). The second refers to the measurement of the performance of companies with multi-criteria methods and the so-called meta-choice problem that always arises in a benchmark multi-criteria analysis that can be synthesized as follows: “how to choose an algorithm to choose?” (Iazzolino et al., 2012; Laise et al., 2015). By putting these two contributions together, we propose a new model for evaluating the performance of knowledge organizations that includes the different components of knowledge capital as inputs and includes the Value Added produced by the institutions among the outputs. By applying this model to the case study of Italian universities, we highlight how the evaluation of the business performance of knowledge organizations is affected by three different meta-choice problems that interact with each other, influencing the obtained results.

The main result we show is that behind rankings there is not a perfect measurement, or in economic terms, a maximization of performance is not feasible (or reachable), owing to the existence of the three meta-choice problems. The first is the meta-choice problem recalled above and investigated by Iazzolino et al. (2012) and Laise et al. (2015) which relates the methodology dimension in Daraio (2017a) and underlies the choice of the algorithm used to compute the ranking. A second meta-choice problem underlies the theoretical dimension of the modeling, called theory in Daraio (2017a), that relates to the following choice: which variables to include in the model or better “how to choose the theory to identify the variables to consider in the model and in the empirical analysis? A third meta-choice problem that arises relates the data dimension in the framework of Daraio (2017a) and consists in the choice and main limitations of the data on which the analyzes are carried out, in other words, what data have to be used in the analysis, and how data problems and limitations affect the empirical analysis?

As we will see in the empirical illustration on Italian universities, these three meta-choice problems are important and are able to affect the rankings of the knowledge institutions considered; furthermore, they interact with each other, as witnessed by the complexity of the evaluation related to its implementation (Daraio, 2017b).

When studying Universities and in general knowledge-based organizations, we have to consider an important element. The main characteristic that drives performance is knowledge and, in particular, the so-called intellectual capital (IC). The inclusion of knowledge and IC in the assessment of performance is not immediate also owing to the data problems related to its measurement.

In this paper, we focus on multidimensional benchmarking analysis applied to key performance indicators (KPIs). KPIs are related, on the one hand, to the IC (divided in the three dimensions of Human Capital, Structural Capital and Relational Capital) and on the other hand, to performances, evaluated in both financial (Revenues, Value added and EBITDA) and non-financial (number of publications and number of patents) terms.

The paper provides several practical implications in all cases in which a ranking has to be assigned to a group of organizations based on performances.

The adoption of a set of criteria is certainly an advantage to avoid mono-criterial or mono-dimensional myopic evaluation. However, this also creates some methodological problems. The paper demonstrates the difficulties of the so-called “implementation problem” in performance measurement, related to the “relativity” (subjectivity) of results of the evaluation process when there are many evaluation dimensions, as is the case in a benchmark context.

2. Complexity of the assessment: the meta-choice problems of knowledge organizations

In this paper, we want to highlight the meta-choice problems that always arise in a multidimensional benchmarking analysis.

The authors of this paper argue that any multidimensional benchmarking evaluation implies the development of a model that concerns the choices made from a theoretical, methodological and empirical point of view (data). See Figure 1 that shows the main dimensions of an assessment which coincide with the meta-choice problems.

From a methodological point of view, a meta-choice problem arises because multi-criteria (or multidimensional) ranking algorithms cannot be selected using a multi-criteria algorithm; the choice of an algorithm is ultimately determined by the subjective preference of the policymaker; and the meta-choice solution to the benchmarking problem is, in accordance with Simon’s satisficing solution, describing a non-maximizing performance measurement methodology.

To perform a benchmark analysis, a set of dimensions in addition to criteria must be chosen. The choice of the main conceptual references and main conceptual dimensions to be considered in a multidimensional benchmark highlights the theoretical meta-choice problem. To decide what are the main conceptual references of the model of the benchmark we pursue, we cannot use a theoretical justification, we need to explicit the subjective preference of the policymaker and/or the analyst who carry out the analysis.

A third meta-choice problem arises when an empirical analysis has to be carried out. A third dimension to consider is data, and the problems of choosing the data, the variables that proxy them, their availability and their quality interact with the two previously described meta-choice problems, showing the complexity of the benchmarking exercises particularly when the focus is on multi-criteria benchmarking analysis applied to a set of knowledge organizations.

Figure 2 illustrates the decision-making problem that managers have to face in multi-criteria benchmarking analysis.

3. Case study: analysis of the performance of Italian universities

In this section, we illustrate the case study. The empirical analysis is based on the analysis of the performance of Italian Universities.

3.1 Sample and data collected

The analyzed sample is constituted of 64 Italian universities. We consider universities of different types and size: 11 mega-universities, 15 large universities, 17 medium-sized universities, 12 small universities, 4 polytechnics, 2 doctoral institutes and 3 schools of advanced study that are part of the Italian higher education system. Data collection considers three years, from 2016 to 2018.

The indicators of inputs and outputs to evaluate the performance of Italian universities will be illustrated in detail in the next section.

3.2 Selected indicators: intellectual capital and performance

This paper proposes a set of indicators especially designed for universities and related to the IC dimension.

The term intellectual capital (IC) was first introduced by John Kenneth Galbraith: the concept of the term incorporated a degree of “intellectual action” rather than “intellect as pure intellect” (Edvinsson and Sullivan, 1996). Dumay (2016) defines IC as the collection of intangible resources, knowledge, experience and intellectual property an organization, community, country or society has and uses to create economic, utility, social and environmental values. The intangible assets and IC constitute the largest proportion of universities’ assets (Ramírez et al., 2011; Secundo et al., 2010). When related to a university, IC is a term used to cover all the institution’s nontangible or nonphysical assets, including processes, innovation, patents, the tacit knowledge of its members and their capacities, talents and skills, the recognition of society, its network of collaborators and contacts, etc. (Ramírez Corcólez et al., 2013).

At an international level, it is generally accepted that there are three basic components of IC: (1) Human capital, (2) Structural capital and (3) Relational capital (Ramezan, 2011; Steward, 1994; Johnson, 1999; Smith and Parr, 2000; Edvinsson and Malone, 1997; Secundo et al., 2017). The three dimensions of IC have a positive and significant influence on organizational performance (Ibarra-Cisneros et al., 2020). The components of university’s IC have been categorized in various ways, although undoubtedly it is the tripartite classification that is most widely accepted in the specialized literature (Secundo et al., 2010; Leither, 2004; Bezhani, 2010; Paloma Sànchez et al., 2009). Specifically, it is possible to read the three components as follows (Ramírez Córcoles et al., 2011):

1. Human capital: The sum of the explicit and tacit knowledge of the university staff (teachers, researchers, managers, administration and service staff) acquired through formal and non-formal education and refresher processes included in their activities.

2. Structural capital: The explicit knowledge related to the internal process of dissemination, communication and management of the scientific and technical knowledge at the university.

3. Relational capital: The extensive collection of economic, political and institutional relations developed and upheld between the university and its non-academic partners, i.e. enterprises, non-profit organizations, local government and society in general.

The input indicators were selected by the authors within the set of indicators generally accepted in the literature (Paloma Sànchez et al., 2009; Córcoles, 2013; Di Berardino and Corsi, 2018; Frutos-Belizón et al., 2019) and are grouped in the three different components of the IC as follows:

Input indicators:

Human Capital (HC)

1. Number of Academic Staff

2. Number of Technical and administrative staff

3. Academic Staff costs

4. Technical and administrative staff costs

Structural Capital (SC)

1. Number of Departments

2. Patents and similar intellectual property rights

3. Licenses and trademarks

4. Scientific equipment

Relational Capital (RC)

1. Grants from others (private)

2. Grants from others (public)

3. Revenues from research projects

The main goals for universities are generally accepted to be the production, diffusion, transfer and preservation of knowledge (Young Chu et al., 2006); for this reason, university performance assessments are defined by the output indicators listed below.

Output indicators:

1. Total Revenues

2. Value Added

3. EBITDA

4. Number of patents

5. Number of spin-offs

6. Number of journal articles

The indicators were selected with the criteria of the feasibility of data gathering and of consistency between universities: most of the indicators can be valued through the items of the university’s income statement and balance sheet, the others through online portals.

Some important theories (Kaplan and Norton, 1996; Sveiby, 1989) suggest that non-financial measures provide a means of complementing financial measures and should also be present at the strategic level of the firm; therefore, we consider both financial indicators on the amount of resources devoted to a given activity and non-financial indicators, such as number of academic staff or number of spin-offs.

4. Results

4.1 Descriptive analyses

On the basis of the discussion and the literature described in the previous section, we identified the variables reported in Table 1 to be considered, respectively, as inputs and outputs to assess the performance of the Italian universities.

The tables reported in Appendix show some descriptive statistics on the data available for the 64 Italian universities over the years 2016–2018. As emerges by inspecting the values reported in the tables, there is a high heterogeneity among the Italian universities considered (high standard deviation values and high interquartile ranges) and most of the dimensions considered of inputs and outputs show skewed distributions (average and median values differ for almost all variables over the whole period). For the variables X11 and Y6, the data are available only for 2018.

The following Tables 2 and 3 show the correlations among inputs and outputs, respectively.

The analysis of the correlations is particularly useful to carry out a preliminary assessment of the relationships among the variables because for the methods that will be used to compute the rankings of the Italian universities, and in particular, for the Data Envelopment Analysis (DEA), the so-called “curse of dimensionality” is really a plague. This means that the algorithms involved in the DEA efficiency score computation require many (even thousands!) of data when the number of variables used is quite high, as is the case here (for further details, see, e.g. Daraio and Simar, 2007).

For this reason, the least correlated variables were chosen, excluding variables that show a correlation greater than 0.9 and were redundant for the analyses. In addition, in the final selection of variables, we considered also the theoretical significance of the indicators. Hence, given the high correlation between variables X1, X2, X6, X7 and X11, and given the theoretical significance of the represented indicators, only X1 and X2 were selected as indicators for personnel. Moreover, given the high correlation between X1 and X2, these two indicators were aggregated into a single variable, “personnel costs” (I1). In addition, X3 and X4 were also aggregated in the variable I4 because they express similar concepts despite the low correlation. X10 was excluded because, despite having a low correlation between the other variables, it has a high percentage of missing information (a percentage of 31%, 60 units out of 192 units). The same kind of reasoning was applied to the outputs. Given the high correlation between variables Y1 with Y3, Y4 and Y5 and given the low correlation between the other variables, Y1 was excluded. Y4 and Y5 have a very high correlation between them and almost the same correlation with the other variables, so we decided to exclude Y5 and use Y4 for our analysis.

At this stage of the analysis, then, we see that the selection of the dimensions of performance to calculate the ranking of universities is influenced by a methodological problem (the curse of dimensionality).

Table 4 shows the variables that will be finally used to calculate the rankings of Italian universities.

The indicator I1, which indicates total personnel cost, is the sum of the variables X1 and X2 in Table 1. By “Scientific Equipment” (I2), we mean the instruments used mainly in laboratories. They relate to scientific and research activities and they may also have a high technological content. Indicator I3 contains the licenses and trademarks; they indicate the granting of rights on goods owned by the granting institution. I4 is the sum of the variables X3 and X4; it represents all external contributions to universities. The I5 indicator indicates the resources obtained by the University from research projects commissioned by external parties.

“Value Added” (O1) is the difference between Production and External costs. O2 represents the number of patents owned by the university. O3 indicates the number of scientific papers published in peer-reviewed journals. “Spin-offs” (O4) are firms founded by academics.

4.2 Methods applied to compute the rankings of knowledge organizations

As described in the introduction, the aim of this work is to show how the three meta-choice problems emerge in the evaluation of the performance of knowledge organizations and influence the results obtained.

The concept of performance we pursue in this paper is multidimensional. Measuring performance means having a representation model of the output/outcome process connected to the inputs (resources) needed to produce it. It also requires the availability of data to apply mathematical–statistical methods to assess performance. Performance measurement methods include quantitative frontier benchmarking methods and multi-criteria methods.

For the comparison of performance results, we chose two types of methods: (1) methods based on the estimation of a multidimensional best-practice frontier, based on multiple inputs and multiple outputs, called Efficiency analysis methods; (2) multi-criteria methods, based on different criteria, specified as benefits and costs, to measure the performance of knowledge organizations, called Multi-Criteria Decision methods (MCDM).

Efficiency analysis methods are based on the estimation of an efficient benchmarking frontier against which to compare the performance of a sample of units. The efficiency scores obtained in an efficiency analysis, based on the estimation of the distance of each unit in the sample from the efficient frontier, allow us to rank the units in the sample according to the performance score obtained. In the literature on efficiency analysis, the nonparametric approach has received a considerable amount of interest in the context of multiple performance measurement, both from a theoretical and an applied perspective. This is mainly because it does not require many assumptions and particularly because it does not need the specification of a functional form for the frontier. Hence, the parameters of the functional form of the frontier do not have to be estimated in this approach, from which the name “nonparametric” approach derives; whereas in the parametric approach, the parameters of the efficient frontier must be estimated. DEA (Charnes et al., 1978; Banker et al., 1984) and Free Disposal Hull (FDH, Deprins et al., 1984) are among the best-known and most applied nonparametric techniques for the measurement of the efficiency in many service activities, including those of universities. DEA uses mathematical programming techniques to estimate a set of efficiency scores that measure the distance of a set of units from an efficient or best-practice frontier. DEA is based on two main assumptions: the convexity and the free disposability of the attainable set. In a DEA setting, it is also possible to choose the returns to scale of the best-practice frontier, considering for instance Constant Returns to Scale (CRS) or Variable Returns to Scale (VRS). In CRS production processes, an increase of 10% of all the inputs produces an increase of 10% of the output, while in VRS production processes, there may be constant, increasing or decreasing returns to scale, admitting the variability of the returns to scale. It is also possible to use instead of DEA, the FDH estimator that assumes only the free disposability of the production set, from which the name Free Disposal Hull derives. An illustration of these different nonparametric efficiency estimators is reported in Figure 3.

Figure 3 shows a simplified efficient frontier estimated using an input factor and an output factor which aggregate, respectively, all the inputs and all the outputs according to the factorial method described in Daraio and Simar (2007, pp. 148–149). DMU is decision-making unit and identifies each observation reported in the plot. CRS, the black line in Figure 3, is the efficient frontier estimated with DEA under the hypothesis of Constant Returns to Scale (CRS). VRS, the dashed line in Figure 3, is the efficient frontier estimated with DEA under the hypothesis of Variable Returns to Scale (VRS). FDH, the gray line in Figure 3, is the efficient frontier estimated with Free Disposal Hull. As the plot shows, we can see that the CRS frontier is the furthest from the cloud of points of the DMUs (the points reported in the graph). On the other hand, the VRS frontier seems to be a closer envelopment of the observations (DMUs), being closer to the cloud of points. However, the VRS frontier relies on the hypothesis of convexity that could be violated by the observed data and, therefore, should be tested before adopting the DEA approach. As Figure 3 clearly shows us, we have three different estimators of the efficient frontier that provide different frontiers. The value of the efficiency scores, which are the distances of each DMU from the best practice efficient frontier, changes according to the selected efficient frontier estimator (CRS, VRS or FDH).

Multi-criteria decision analysis is a discipline aimed at supporting the decision-making process when there are numerous evaluations, allowing a compromise solution to be obtained in a transparent way. This methodology allows the decision maker to analyze and evaluate different alternatives, monitoring their impact on the different players in the decision-making process. There are various methods for multi-criteria analysis (Vincke, 1992; Figueira, 2005). MCDM methods offer the possibility of finding out satisfactory measures of performance that provide a balance between multiple criteria offering a solid and balanced support to the decision-making process.

The MCDM methods considered in this paper are those implemented in the R package “Multi-Criteria Decision Making Methods for Crisp Data” by Ceballos Martin (2016). These methods consider two kinds of dimension: benefits (that we will consider equivalent to the outputs of the efficiency analysis) and costs (that we will consider equivalent to the inputs of the efficiency analysis) and differ in the specification of different technicalities, including for instance the normalization adopted. The MCDM that will be applied with their labels (that will be used in the following to illustrate the results) are the following:

1. Multi-Objective Optimization by Ratio Analysis labelled as RSM (Brauers et al., 2010).

2. Multi-Objective Optimization by Reference Point labelled as RPM (Brauers et al., 2010).

3. Multiplicative Form labelled as MFM (Brauers et al., 2010).

4. Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) Method with the linear transformation of maximum as normalization labelled as TSL (Garcia Cascales et al., 2012).

5. Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) Method with the vectorial normalization procedure, labelled as TSV (Hwang et al., 1981).

6. Weighted Product Model, labeled as WSM (Zavadskas et al., 2012).

7. Weighted Sum Model, labeled as WPM (dZavadskas et al., 2012).

4.3 Comparative analysis on the obtained results

Both efficiency analysis methods and MCDM then provide as a result a ranking of the knowledge-based organizations according to a multidimensional perspective.

The comparative analysis on the rankings of the Italian universities will be based then on the calculation of different rankings according to a few variants of the efficiency analysis and MCDM methods. After that, we will compute the Spearman ranking correlations among the results obtained to check how rankings vary according to the methods applied.

Moreover, to run a balanced comparative analysis among all the methods, we choose the same vector of weight for all the dimensions considered. For the same purpose (balanced comparison), all the scores have been normalized to obtain comparable values comprised between 0 and 1.

Before comparing the results obtained by efficiency analysis and MCDM, a natural question arises: that is, what method to choose among the efficiency methods introduced earlier? We know that DEA relies on the convexity assumption while FDH does not rely on it. To answer to this question, we test for the convexity assumption, by applying a recently introduced test by (Kneip et al. (2016) and Daraio et al., 2018) and we did not accept the convexity assumption on our data. This means that the application of DEA methods is not appropriate for our dataset, and in the following of the analysis, we will use only the FDH efficiency scores.

There is an additional warning we have to take into account. The FDH estimator of efficiency scores is determinist by nature. This means that all the deviations observed from the efficient frontier are attributed to inefficiency, so no noise is allowed. As a consequence, the efficiency scores calculated by FDH suffer from the influence of outliers and/or errors in the data. For this reason, before the comparison with MCDM, we considered also a robust nonparametric efficiency method that is not influenced by outliers: namely, an order-m efficient frontier estimator, where m is the number of random peers selected to compute the robust frontier which does not envelop all the DMUs leaving out the most extreme and outliers observations (for more details, see Daraio and Simar, 2007).

But a question may arise now to the reader that is the following. Of course, by applying different methods, we obtain different results and so different values for the efficiency scores of the Italian universities; but are those units that are on the top of one ranking also on the top of the other rankings? In other words, is the rank correlations among the efficiency scores high or low?

To answer this question, we computed the Spearman’s rank correlations among the efficiency scores obtained by applying FDH in an output orientation (which means given the inputs we look at the maximum production of the outputs, indicated as FDH O in the following) with the efficiency scores obtained by applying an order-m frontier estimation with m = 25 and with m = 100.

A Spearman’s rank correlation value close to 1 means that the two efficiency scores calculated on the basis of two methods, although they show different values, correspond to the same ranking of the universities analyzed. A rank correlation close to −1 indicates that the efficiency scores calculated with a method are almost the inverse to the order of those calculated with another method, meaning that those universities that are on the top of a method are on the bottom of the other and vice versa. Intermediary values of the rank correlation show varying level of correlations among the rankings obtained by the different methods.

The results of the rank correlation calculated between FDH O efficiency scores and Order-m with m = 25 is 0.95 while the rank correlation obtained between FDH O and order-m with m = 100 efficiency scores is 0.99.

On the basis of the high values of the rank correlations, we can consider FDH O efficiency scores as not affected significantly by outliers in the data and in the following we will use only FDH O in the comparison with MCDM methods.

The results of the comparison among FDH O efficiency scores and MCDM methods are reported in Figure 4, which shows the boxplots of normalized FDH O and MCDM scores calculated on the sample of Italian universities.

A boxplot is a graphical representation used to describe the distribution of a sample using simple dispersion and position indices. It is represented by a rectangle divided into two parts, from which two segments come out. The rectangle (the “box”) is delimited by the first and third quartiles and divided inside by the median. The segments are delimited by the minimum and maximum of the values.

By inspecting Figure 4, we can see that the scores obtained by the different methods are all different.

Table 5 reports Spearman’s correlations calculated among the normalized scores obtained by the different methods implemented.

Inspecting the values reported in Table 5, we can observe that the ranks or positions obtained by the units according to the different methods differ, and so the choice of the method to apply should be carefully discussed before using the obtained ranking for decision-making purposes.

A final meta-choice problem that arises in the evaluation of the rankings of knowledge organization, relates to the choice of the theory, considering how the results are affected including or excluding some variables. For illustrating this meta-choice problem, we analyze what happens if we include in the FDH O model as inputs all intellectual capital components (namely, HC, SC and RC) or only each of them separately, keeping the outputs constant (the same as the previously described overall model). The results are reported in Figure 5 and Table 6. From Figure 5, we see that choosing all the components of the Intellectual capital as inputs or selecting a component of it affects the results obtained (the boxplots are all different). When considering the HC and RC, we observed that there are several outliers (the points that fall outside the boxes) that differ from the bulk of the observations. Spearman’s correlations reported in Table 6 confirm the large variation of the results obtained showing very low rank correlations. Again, these results show that the third meta-choice problem is in place and the selection of the variables that have to be included in the analysis should be carefully motivated by existing theory (the literature). In our case study, the choice of considering all the different components of IC is motivated by the review of the literature carried out and also confirmed by the results reported in Table 6 and Figure 5, which show how each component of the IC plays a role (determining different results) and for this reason it is advisable to include all the components of the intellectual capital in the empirical analysis.

5. Discussion and conclusion

In this paper, we tackle the issue of knowledge visualization and its connection with performance measurement from an epistemological point of view. Following Carson (2020) and more generally, the history of quantification on which Carson relies on, we consider quantification and measurement not just as technical questions but show their relevant implications on the management decision-making of knowledge-based organizations. This is because quantification and measurement produce empirical results, such as rankings, that can be used to inform and support the decision-making process of Knowledge organizations. If the results obtained by the analysis are not carefully considered in terms of reliability and robustness, they can provide an unreliable and biased support to the decision-making process.

The main conclusions of this paper are the following:

1. In the evaluation of knowledge organization (as universities are), when the evaluation is based on different genuine criteria and on a set of different dimensions including IC, three meta-choice problems arise and interact among each other. Multi-criteria ranking algorithms cannot be selected using a multi-criteria algorithm (methodological choice); so, they rely on the subjective choice of the analyst/policy maker who has to choose also the conceptual background of the benchmarking (theoretical choice) addressing the empirical issues that arise (data choice).

2. The choice of an algorithm, of the conceptual reference and of the data is ultimately determined by the subjective preference of the analyst/policy maker who should be explicitly described, highlighting the “fitness for purpose” strategy (satisficing principle) followed, inevitably based on the compromise choices made to address the existing meta-choice problems.

Our proposal or solution to the benchmarking problem is in accordance with Simon’s satisficing solution, describing a non-maximizing performance measurement methodology (Simon 1955, 1978, 1997). It may be worth emphasizing that Simon’s point of view is adopted by the Managerial Accounting multi-criteria approach: “we believe however that achieving satisfactory profit is a better way of stating corporation goals” (Anthony, 1966). As is well known, such a perspective has a long tradition in managerial and accounting literature (Anthony, 1966; Cyert and March, 1963; Drucker, 1966; March and Simon, 1958; March, 1966a, 1996b). This approach is strictly related to the multidimensionality in measuring firm performances. There is a traditional way of measuring the performance of an organization that mainly focuses on value creation for the shareholder (shareholder point of view) (Jensen, 2002). According to this approach, the only variable to be considered is related to profit and ultimately to dividend for shareholders. But in the modern enterprise, the performance cannot be measured only on one dimension. The approach based on the Balanced Scorecard (Kaplan and Norton, 1996) and the theme of Sustainability point out that the issue of measuring organizational performance is a genuine multi-criteria problem. A sustainable strategy has to create value not only for the shareholder, but also for the other stakeholders such as, for example, the employees and the external environment. It is necessary to switch from the shareholder point of view to the stakeholders point of view, in which many actors have to be considered for correctly evaluating the performance of an organization. The evaluation of the organizational performance cannot generally be conducted by means of a unique criterion but considering a multi-criteria approach (Iazzolino et al., 2012; Laise et al., 2015).

Our suggestion to the analyst who conducts performance assessments is therefore to be transparent with stakeholders (including policy-makers and managers of universities that use rankings), describing all the crucial meta-choices that underlie their analyses and highlighting the role that they have on the results. This behavior, defined “technologies of humility” by Jasanoff (2007) could be achieved through a checklist as proposed by Daraio (2019), according to which, the analyst describes all the choices made in the analysis and the impact that these choices have on the results obtained, possibly identifying lines for further improving the analyses. This approach corresponds to the awareness of having achieved a “satisficing” result in terms of rankings of the institutions, à la Simon, which considers the hypotheses and compromises made, rather than an objective measure of the institutions that may fit all the purposes.

This research has some limitations that we suggest addressing in future studies from both a theoretical and an empirical point of view.

From a theoretical perspective, the authors have already started to explore deeply the relationships between IC and performance of knowledge organizations to obtain a systematic review of the topic. The review can be carried out from a strictly scientific/academic point of view, and so mainly analyzing journal papers, but also by considering the most-used methodologies in managerial practice. It would be interesting to map the existing information and relationships between IC and Performance with a focus on knowledge organizations, including Universities, but not limited to them.

From the empirical point of view, the main issue will be related to the sample used for the analysis. Our research was carried out using a sample of Italian universities. The sample could be enlarged to include other Universities, also belonging to other countries.

Figures

Figure 1

Main dimensions of the meta-choice problems of knowledge organizations

Figure 2

An illustration of the decision-making problem with existing meta-choices

Figure 3

Plot of the efficient frontiers estimated according to different methods

Figure 4

Boxplots of normalized FDH and MCDM scores calculated on the sample of Italian universities

Figure 5

Boxplots of results obtained by the different FDH models: FDH O includes all IC inputs, i.e. HC, RC and SC. FDH O_HC includes as input-only HC as IC. FDH O_RC includes as input-only RC as IC. FDH O_SC includes as input-only SC as IC

Table 1

Inputs and outputs selected according to the literature

	Label
Input
Technical and administrative staff costs	X1
Academic staff costs	X2
Grants from others (private)	X3
Grants from others (public)	X4
Revenues from research projects	X5
Number of technical and administrative staff	X6
Number of academic staff	X7
Scientific equipment	X8
Licenses and trademarks	X9
Patents and similar intellectual property rights	X10
Number of departments	X11
Output
Total revenues	Y1
Number of patents	Y2
Number of journal articles	Y3
Value added	Y4
EBITDA	Y5
Number of spin-offs	Y6

Table 2

Input correlations matrix

	X1	X2	X3	X4	X5	X6	X7	X8	X9	X10	X11
X1	1
X2	0.962	1
X3	0.66	0.721	1
X4	0.435	0.44	0.292	1
X5	0.539	0.586	0.723	0.103	1
X6	0.968	0.943	0.622	0.43	0.477	1
X7	0.933	0.968	0.716	0.393	0.592	0.961	1
X8	0.49	0.521	0.433	0.208	0.292	0.48	0.502	1
X9	0.204	0.242	0.354	−0.001	0.076	0.209	0.233	0.13	1
X10	0.465	0.448	0.461	0.026	0.493	0.419	0.448	0.261	0.259	1
X11	0.906	0.905	0.586	0.443	0.381	0.948	0.93	0.517	0.212	0.369	1

Table 3

Output correlations matrix

	Y1	Y2	Y3	Y4	Y5	Y6
Y1	1
Y2	0.725	1
Y3	0.931	0.675	1
Y4	0.996	0.713	0.923	1
Y5	0.921	0.677	0.835	0.932	1
Y6	0.488	0.584	0.479	0.483	0.405	1

Table 4

Inputs and outputs finally chosen for the empirical analysis

Input variable	Output variable
I1: Total cost of employees	O1: Value added (VA)
I2: Scientific equipment	O2: Number of patents
I3: Licenses and trademarks	O3: Number of journal articles
I4: Grants from others (public + private)	O4: Number of spin-offs
I5: Revenues from research projects

Table 5

Spearman’s correlations of normalized FDH and MCDM methods’ scores

	FDH O	WSM	WPM	TSV	TSL	RSM	RPM	MFM
FDH O	1
WSM	0.55	1
WPM	0.59	−0.03	1
TSV	0.39	−0.05	0.84	1
TSL	0.54	1	−0.06	−0.07	1
RSM	0.54	0.09	0.87	0.95	0.07	1
RPM	−0.21	−0.4	−0.22	−0.46	−0.4	−0.43	1
MFM	0.59	−0.03	1	0.84	−0.06	0.87	−0.22	1

Table 6

Spearman’s correlations calculated on the results obtained by the following different FDH models

	FDH O	FDH O_HC	FDH O_ RC	FDH O_ SC
FDH O	1
FDH O_HC	0.47	1
FDH O_ RC	0.43	0.26	1
FDH O_SC	0.32	0.19	0.12	1

Note(s): FDH O includes all intellectual capital inputs, i.e. HC, RC and SC. FDH O_HC includes as input-only HC as intellectual capital. FDH O_RC includes as input-only RC as intellectual capital. FDH O_SC includes as input-only SC as intellectual capital

Table A1

Descriptive statistics on the analyzed data, 2016

2016	Average	Standard dev	Min	First quartile	Median	Third quartile	Max
X1	30886990.63	30068500.65	839241.16	10699106.57	22206621	38263916.35	156479849.6
X2	72862854.9	64905310	2183048.99	23132656.26	60035952	89545413.62	303736781.2
X3	3239643.788	4327921.332	0	599330.085	1933586.21	3184376.635	19471787.24
X4	3534592.24	5737047.783	0	613198.795	1372930.095	3976555.37	33445929.37
X5	4100221.247	6942952.719	0	625289.37	1520443.495	4409778.095	45069339.49
X6	813.09375	775.6293035	18	274.25	586	1058.75	4101
X7	1004.8125	901.2046942	30	344.25	825.5	1239.25	4340
X8	4980193.873	6074817.025	0	927085.8975	2691301.345	6967648.993	35481238.27
X9	4451372.209	32115245.54	0	0	10035.1	109927	254740019
X10	134716.35	238669.3073	0	0	19857.02	163518	1144351
Y1	182588052.6	165433466.4	0	58839021.33	139100272.6	225258162.3	794318158
Y2	202.90625	223.1282441	0	57	150	243.75	1295
Y3	1880.65625	1808.456601	0	593.25	1352	2435.75	8002
Y4	132016143.7	120549687.8	0	45739328.5	98030140	171780055	579161026.9
Y5	29934612.82	29664840.76	0	10627828.62	20613848.08	40182185.72	135099933

Table A2

Descriptive statistics on the analyzed data, 2017

2017	Average	Sd	Min	First quartile	Median	Third quartile	Max
X1	30484819.09	29008906.59	906129.88	10579975.1	21953933.91	40933881.23	154763171.3
X2	73602877.01	66358404.55	1950996.46	23318411.22	58881091.09	91474051.3	297412285.9
X3	2995596.73	3633393.984	6903.64	535801.715	1904430.39	3825308.385	14645497
X4	3190562.355	5610717.58	0	437803.5025	1182479.53	3602470.943	32948685.92
X5	4185693.309	7579828.311	0	699837.0075	1878066.26	4283587.978	51184764.48
X6	801.703125	763.769987	18	265.75	566.5	1038.5	4102
X7	998.34375	896.5639385	30	334	802	1253.5	4145
X8	4835263.797	5982081.668	0	964965.27	2926961.515	6009921.6	36838861.6
X9	4383974.196	31864034.58	0	312.6325	12582.18	113980.25	254740019
X10	139887.1169	265470.2285	0	0	18706.215	120588.65	1403037
Y1	189353686.7	168924388	6582896.96	61451593.75	145881253.1	229751394.5	769644208.1
Y2	210.5	236.9207221	0	58	156.5	253.5	1413
Y3	1911.453125	1838.670387	0	601.25	1359	2542	7943
Y4	135742018.1	120992899	−2781198.77	44578909.92	103411672.7	170337236	568487762.8
Y5	31654321.99	28949335.87	−5638325.11	10288626.56	24876372.5	40803251.35	118842044

Table A3

Descriptive statistics on the analyzed data, 2018

2018	Average	Sd	Min	First quartile	Median	Third quartile	Max
X1	31540692.23	29185518.26	1042861.51	10468668.5	22096770	43095741.26	157343178.1
X2	70838356.1	62858805.98	1828979.2	23861065.57	53084760.1	86677668.54	291157122.4
X3	3037705.467	4272521.389	4460	476008.335	1853033.48	3270972.725	23245571.07
X4	2578927.571	4075890.991	0	385022	1274234.22	3249584.045	22105574.39
X5	4667227.099	8519516.767	0	801773.87	1817319.45	4858339.455	59760757.9
X6	788.765625	737.8925964	23	262.75	559	1027.5	3906
X7	1003.359375	902.1298321	34	340.25	776	1281.25	4090
X8	4599266.341	6229324.029	0	753196.14	3101016.52	4852363	40417510.15
X9	158969.2489	371274.466	0	150.4325	15835.35	112703.5775	2008491.66
X10	139350.84	278966.5668	0	0	13600.205	141837	1370355
X11	12.171875	9.692993194	1	6	10	16	58
Y1	184681310.3	169186477.3	0	57003842.07	139625754.6	231277795.3	783678965.1
Y2	216.265625	248.6477232	0	60.25	163	260.25	1522
Y3	1858.8125	1783.683276	0	574.75	1352.5	2349.75	7517
Y4	130674081.4	119728940	−2894887.35	44163616.23	98147142	165589621	581301790
Y5	29894705.75	30000559.37	−11126794.83	9396363.393	21331117.12	40360152.5	132801489.5
Y6	18.35483871	15.17666109	0	7	15	27	70