Big data analytics in innovation processes: which forms of dynamic capabilities should be developed and how to embrace digitization?

Abstract

Purpose

The purpose of this paper is to analyze, from a dynamic capabilities perspective, the role of big data analytics in supporting firms' innovation processes.

Design/methodology/approach

Relevant literature is reviewed and critically assessed. An interpretive methodology is used to analyze empirical data from interviews of big data analytics experts at firms within digitally related sectors.

Findings

This study shows how firms leverage big data to gain “richer” and “deeper” data at the inter-sections between the digital and physical worlds. The authors provide evidence for the importance of counterintuitive strategies aimed at developing innovative products, services or solutions with characteristics that may initially diverge, even significantly, from established customer/user needs.

Practical implications

The authors’ findings offer insights to help practitioners manage innovation processes in the physical world while taking investments in big data analytics into account.

Originality/value

The authors provide insights into the evolution of scholarly research on innovation directed toward opportunities to create a competitive advantage by offering new products, services or solutions diverging, even significantly, from established customer demand.

Citation

Capurro, R., Fiorentino, R., Garzella, S. and Giudici, A. (2022), "Big data analytics in innovation processes: which forms of dynamic capabilities should be developed and how to embrace digitization?", European Journal of Innovation Management, Vol. 25 No. 6, pp. 273-294. https://doi.org/10.1108/EJIM-05-2021-0256

1. Introduction

Digitization is profoundly reshaping the way firms think and go about creating a competitive advantage (Baden-Fuller and Haefliger, 2013; Huarng et al., 2015; Lanzolla and Giudici, 2017) not only in the so-called “digital world” (e.g. Alberti-Alhtaybat et al., 2019) but also in the “physical” world (i.e. “real-traditional world,” Chen and Zhang, 2014; Hartmann et al., 2016; Tian, 2017). For example, digitization increasingly opens up opportunities for firms to conceive and create innovative business models in two-sided markets (Caputo et al., 2019; Rochet and Tirole, 2006; Garzella et al., 2020) with interconnections between two or more customers that might or might not make monetary payments (i.e. “users,” see Baden-Fuller et al., 2018; Erevelles et al., 2016; Tiago and Verìssimo, 2014).

In the digital transformation, automated systems equipped with automatic data exchange and technology capabilities characterized by “self-monitoring, analysis and reporting technology (SMART)” (Mashhadi et al., 2018, p. 1108) play a key role (e.g. Fiorentino et al., 2020). Among smart technologies (Lu and Weng, 2018), prior studies have emphasized the growing role of big data analytics as a cornerstone of firm performance and competitive success (Ferraris et al., 2018; Wamba, 2017; George et al., 2014) and argued that a firm's capacity to leverage them can be a powerful dynamic capability (Giudici and Reinmoeller, 2012; Teece, 2018; van Rijmenam et al., 2018). Big data analytics are important because they enhance a firm's ability to connect technology and customers/users (Brown et al., 2011; Dobusch and Kapeller, 2018), thus enabling the aggregation of a large amount of data about technology development (Papadopoulos et al., 2017) and predicted customer demand (Bresciani et al., 2018).

However, despite the promises of big data analytics for strategic innovation, academic research remains inconclusive (Snihur and Wiklund, 2018). A specific literature stream analyzing the importance of capabilities and providing further theoretical and empirical elaboration on issues related to technology, strategic management and innovation in today's digitally enabled networked contexts (Kouropalatis et al., 2019; Mikalef, 2019) have developed, yet still, little is known about how firms develop and implement digital innovative strategies, and many firms have been left in the dark about how to invest effectively in big data analytics capabilities to drive their innovation agenda (Bean, 2016). The lack of capabilities or awareness of benefits often prevents the adoption of big data analytics in strategic decision making by executives (Merendino et al., 2018).

Scholarly research on innovation, in particular, has traditionally juxtaposed views emphasizing the role of science and technology inputs (i.e. the “technology-driven” or “technology-push” perspectives, see, e.g. Rosenberg, 1982) and those highlighting the importance of market and user-driven features (i.e. the “market-driven” or “demand-pull” perspectives, see, e.g. Schmookler, 1966). From a “technology-push” perspective, big data analytics are part of the vast family of digital technologies, and the innovation process is increasingly relying on this technology (Nambisan et al., 2017). By contrast, from a “demand-pull” perspective, scholars highlight the role of big data in better understanding customer needs and in firms' innovation processes (Trolio et al., 2017). While calls for more integrative views exist (Brem and Voigt, 2009), a coherent understanding is still lacking on how firms can strategically master the innovation process in a way that strengthens their technology base while simultaneously expanding market demand. Moreover, new insights from studies analyzing the relationships between big data and innovation processes should favor answers to questions such as the following (see, for a review, Kouropalatis et al., 2019): Which capabilities are required to embrace a successful digital transformation based on big data analytics? Which capabilities are critical in the development of innovation processes? How and to what extent is digital transformation changing the innovative processes of firms in terms of new products and services? In this study, we begin to address this gap by investigating the strategic utilization of big data analytics to innovate in the physical world within a set of leader firms by adopting the lens of dynamic capabilities. Specifically, we explore the following research question: how should dynamic capabilities be developed by firms and in which forms to embrace digital innovation?

This research develops an empirical analysis based on a field study capturing the perceptions of key decision makers at firms within digitally related sectors. We adopted an interpretive qualitative approach (Gioia et al., 2013; Lukka and Model, 2010), leveraging data from a variety of sources – interviews, secondary and archival – to contribute to research in strategy and innovation management in three main ways. First, this study extends previous investigations by revealing how innovation processes based on big data analytics differ markedly from traditional paths of innovation. Second, our findings suggest that big data analytics play an important role as the bridge between research on “technology-push” and “demand-pull” innovations. Third, this study details the dynamic capabilities required to overcome this juxtaposition between “technology” and “demand” research and organizes them into four main types. Our findings also offer useful implications to help practitioners incorporate investments in big data analytics as part of their competitive strategy at the intersections with the physical world.

2. Theoretical background

The pioneering study by Teece et al. (1997) defines the concept of “dynamic capability” as a firm's “ability to integrate, build, and reconfigure internal and external competencies to address rapidly changing environments” (Teece et al., 1997, p. 516). Starting with this seminal definition, dynamic capabilities have been investigated from several perspectives and considering specific issues by management studies (for reviews, see Giudici and Reinmoeller, 2012; Kouropalatis et al., 2019).

From the dynamic capability perspective, firms can identify and respond to environmental threats and opportunities by quickly modifying their resources and processes to sustain their competitive advantage over time (Helfat and Peteraf, 2009; Lin and Wu, 2014; Chatzoglou and Chatzoudes, 2018). However, the firm's proactive ability to reinvent existing and novel knowledge requires the selection and management of several sources and varied amounts of information (Merendino et al., 2018; Côrte-Real and Ruivo, 2017). In this vein, recent studies argue that big data analytics can be considered a promising tool to capture weak signals from consumers and use them to predict market and consumer trends (Erevelles et al., 2016, Day, 2014; Reeves and Daimler, 2011; Rialti et al., 2019). Specifically, these studies argue that firms using novel insights extracted from big data to understand unmet consumer needs possess enhanced dynamic capabilities (Wernerfelt, 2014). Big data provides new insights that can be used to improve dynamic capabilities by helping corporate decision-makers to respond to current dynamic environmental trends (Erevelles et al., 2016; Eisenhardt and Martin, 2000). When successfully exploited, the consumer insights obtained from big data facilitate the development of a dynamic business model able to improve firm performance and the value creation process (Bharadwaj et al., 2013; Ferraris et al., 2018). Moreover, scholars have observed that the development of dynamic capabilities supported by big data analytics may trigger the innovation process of firms (Kouropalatis et al., 2019; Teece et al., 2016). Indeed, the dynamic capabilities framework emphasizes the importance of systematically undertaking entrepreneurial innovation as the cornerstone of long-term corporate strategy (Teece et al., 1997). In this vein, successful corporate strategies are based on the ability to identify innovative paths for consolidating and developing new competitive advantages (Levine et al., 2017); firms can face challenges by leveraging high-quality large data to sustain decision-making processes (Davenport et al., 2012; Choi et al., 2017).

In this sense, we argue that the dynamic capabilities framework is a powerful lens for creating, implementing and transforming business models in the digital economy (Karimi and Walter, 2016; Teece, 2018; Teece and Linden, 2017; Balusamy et al., 2017; Quinton and Simkin, 2017; Warner and Wäger, 2019). However, according to recent calls, there is still a need to properly clarify the relationship between dynamic capabilities, big data analytics and digital innovation processes (Dixon et al., 2014; Prange et al., 2018).

2.1 Big data analytics

High velocity, volume and variety distinguish big data and big data analytics, which relies on the creation, storage and use of these data (e.g. Lanzolla and Giudici, 2017). In the current context, the gradual increase in the strategic relevance of information management has moved the focus of firms towards new sources of data such as the web (Barton and Court, 2012; Vitolo et al., 2015). The web has quickly grown to represent the main source of information for firms that, in response, have developed many information systems, models and “machines” to maximize its potential (Vahn, 2014; Balsmeier et al., 2018).

The extent of available information has facilitated the establishment of new procedures for big data analytics (Mashhadi et al., 2018; Carillo et al., 2019). “Intelligence” models have been redefined to attribute increasing relevance to the web dimension and the strategic perspective (Reinmoeller and Ansari, 2016; Lanzolla and Giudici, 2017). Based on the identification of the main dimensions of analysis, information management models and tools have evolved to satisfy the need for strategic decision support (Kuosa, 2011; Garzella and Fiorentino, 2014; LaValle et al., 2011). To achieve a comprehensive and systematic view, scholars have integrated contributions from several managerial disciplines that approach this issue in different ways (Wagner, 2004; Popovic et al., 2012).

In this path, innovation capability has emerged as pivotal (Gobble, 2013). Across several literature streams, scholars have underlined the importance of data analysis and management in supporting creativity and innovation processes (Markides and Anderson, 2006; Olszak and Kisielnicki, 2016; Perry-Smith and Mannucci, 2017). Thus, studies have emphasized the need to elaborate millions of data points to improve customer satisfaction and to develop new ways to approach consumers, creating innovative products, processes or business models (Levine et al., 2017; Shipilov et al., 2017; Mahmoud et al., 2018).

Furthermore, scholars have studied how the analysis of consumer behavior can benefit from big data analytics (Hofacker et al., 2016; Kannan, 2017). Predictive analytics should be useful for developing insights into both consumers' and users' behavior (van Rijmenam et al., 2018; Lu and Weng, 2018). Specifically, big data analytics can support decision makers by helping them understand current and future customer needs, market demand and market trends (Coussement et al., 2015; Jin et al., 2016; Peteraf et al., 2013; Bresciani et al., 2018).

Some scholars suggest that the key success factors in product innovation processes using big data are related to the acceleration of the innovation process, customer connections and the development of an innovation ecosystem (Zhan et al., 2017; Agostini et al., 2019). Moreover, big data analytics can help firms to speed up new product innovation processes and bring down their cost (e.g. Zhan et al., 2017; Wamba et al., 2017). Other studies have highlighted that innovation with pervasive digital technologies reveals relevant traits such as (1) the importance of digital technology platforms, (2) the emergence of distributed innovations, and (3) the prevalence of combinatorial innovations (e.g. Yoo et al., 2012). Indeed, research has shown that high data structuring can present risks if not properly managed, such as the construction of cognitive models that can potentially promote homogenization and limit, rather than stimulate, creativity and innovation (e.g. Choi et al., 2017; Kwon et al., 2014).

This trend is pushing scholars and practitioners towards an in-depth investigation of new approaches based on business analytics, predictive analysis and big data management models to support innovation processes (Vahn, 2014; Carillo, 2017). A need also exists to consider the diffusion of rich data and deep data concepts in addition to big data concepts (Lindstrom, 2016).

2.2 Innovation sources

The “technology-push” and “demand-pull” debate is central in innovation source studies (Nemet, 2009). In the former, the innovation process is mainly the result of new discoveries, inventions and technologies that must be affirmed in the market (Dosi, 1982; Huarng et al., 2015). The latter, instead, argues that the needs of consumers, who are the end users of the products, provide input and stimulate the innovation process (Rosenberg, 1969).

Traditionally, the innovation development model has been structured according to a “technology-push” logic in the literature. This process, developed in the 1950s and dominant until the early 1960s, considers innovation to be the result of a more or less linear process that starts with scientific discovery, continues with R&D, and ends with the creation of industrial technological developments that become new innovative processes and products to affirm in the market (Freeman, 1974). The “technology-push” model therefore recognizes the supremacy of technological progress as the main driver of the evolutionary phenomenon (Rosenberg, 1982). The innovation process arises from the availability of new technologies, and only in the later stages do firms analyze how to meet market needs with these technologies (Mowery and Rosenberg, 1979). It is therefore the firm that promotes innovation through the development of its offer, realized with the support of major direct and indirect investments in research and development that aim to make innovations consistent with the firm's strategic goals. Only later and in the background is there a need to investigate the possible relationship between these innovations and market needs.

In contrast, since the mid-1960s, a different “demand-pull” approach has been developed (Schmookler, 1966). This perspective argues that market demand plays a key role in innovative processes, while technology remains in the background (Myers and Marquis, 1969). Technology knowledge alone is not sufficient to stimulate innovation. The innovative process, understood as the result of market stimuli, reverses the sequence of phases in the “technology-push” model. The primary input of the “demand-pull” process is the identification of market needs. Subsequently, there are attempts to satisfy those needs through technological innovations that place new products on the market (Eggers et al., 2017). Therefore, firm innovations are driven by the need to satisfy market demand, looking to grasp – at best anticipate – market trends to direct and speed the innovation process and acquire new positions of competitive advantage (Chaffey and Ellis-Chadwick, 2016).

The juxtaposition between “technology-push” and “demand-pull” is useful for identifying where the stimuli for innovation arises and for understanding the importance, on the one hand, of focusing on scientific research and technological development activities while, on the other, analyzing the external environment to identify opportunities linked to latent needs (Taylor, 2008; Di Stefano et al., 2012). To date, debate on the primacy of one or the other vision is shifting focus to instead consider ways to integrate these perspectives to fully understand the sources of innovation processes.

3. Methodology and research design

The theoretical background shows that big data analytics are increasingly pivotal to strategic and innovation management decision-making (Chen et al., 2012; La Valle et al., 2011). Big data analytics and related knowledge management are diffusing rapidly as a factor central to firm innovation culture and values (McAfee et al., 2012).

Starting with this premise, we aim to cover a relatively unexplored area of research. Studies are lacking that attempt to provide an in-depth investigation of the strategic utilization of big data analytics for innovation by adopting the lens of dynamic capabilities. Accordingly, we wish to shed light on the dynamic capabilities needed to support the role of big data analytics and their positive impact on firms' innovation processes, in general, and digital innovation specifically (Huesig and Endres, 2019).

To overcome this gap, we adopt an interpretive methodology (Gioia et al., 2013; Lukka and Model, 2010). This well-established approach emphasizes lived experience in management and social studies and is not epistemologically concerned with sampling a specific population but instead focuses on theoretical categories (Eisenhardt et al., 2016; Gephart, 2004). We use semi-structured interview data together with a prior literature review to capture experts' preferences and suggestions related to big data in innovation processes and to investigate which forms of dynamic capabilities experts believe firms should develop and how to effectively embrace digital innovation.

3.1 Respondent selection

To select respondents, we used the snowballing technique (Miles et al., 2014), starting with a set of initial contacts and then asking them for referrals of informants with adequate seniority and knowledge in other firms. We identified a comprehensive group of 25 experts in big data analytics among firms in leading positions in digital-related industries. All experts – shown in Table 1 – were in senior management roles in large international firms from a variety of industries, with complementary expertise, and this allowed us to observe the phenomenon from two perspectives: that of firms developing big data analytics tools and that of firms using them. In line with our interpretive methodology, this group of experts was not intended to represent a representative sample of the population but to support theoretical sampling (Eisenhardt et al., 2016; Gephart, 2004).

3.2 Data collection

The interview instrument, which included guiding questions, was developed based upon the research question, the literature review and our experiences. The interviews were intended to encourage participants to describe their feelings and their cognitive management of the topic, their efforts to manage big data analytics in innovation processes and their own lived experiences.

The interviews included open-ended questions about the “digital world,” big data analytics and innovation processes. Interviews (face-to-face or by telephone) lasted 30–60 minutes and were collected in 2 (second round to seek external validation and refinement) rounds between the end of 2019 and the beginning of 2020. We asked for the informants' view, perceptions, suggestions and practices and, especially, for their experience with innovation processes and big data analytics in the previous five years. The semi-structured nature of the interviews gave them space to freely offer descriptions considering their strategic, and sometimes “visionary,” inclinations regarding the future of digital innovation.

Although the interviews were developed around interactive discussions (Huff and Jenkins, 2002), we started with the following questions:

1. In the digital era, what are the key capabilities and the main tools available for use with the web to define successful corporate and business strategies?

2. What are the main risks when using big data analytics in defining successful strategies and innovation processes?

3. How can big data analytics lead to the redefinition of innovation processes?

4. How can big data analytics lead to the redefinition of relationships between firms and customers and between technology and firms?

5. What are the main issues related to managing the amount of data provided by the web?

6. What are the evolutionary perspectives and dynamics of the relationships between strategies, innovation paths and big data analytics?

3.3 Data analysis

After transcribing the interviews, qualitative data were analyzed using thematic qualitative coding techniques and NVIVO software (O'Kane et al., 2019); several readings and iterations were conducted to highlight frequently occurring themes and patterns and retain illustrative quotations (Miles et al., 2014). Definitions and themes were partly informed by the existing literature and partly emerged from in vivo data (Eisenhardt et al., 2016). We initially developed a tentative coding scheme and identified grounded categories and subcategories related to big data analytics and innovation processes (Guest et al., 2012; Titscher et al., 2000). Then, we developed a coding agenda and examined and interpreted the material by going back and forth between the data and the literature until we achieved a stable degree of triangulation (Graebner et al., 2012; Jonsen and Jehn, 2009). Finally, we submitted our findings to informants to seek external validation and refinement, in line with prior interpretive studies (e.g. Giudici et al., 2018). This final step was important to establish the trustworthiness of our interpretation (Lincoln and Guba, 1985).

4. Findings and discussion

Our explorative research aimed to provide a comprehensive view of ways to use big data analytics to drive firms' strategic innovation processes.

First, evidence from interviews, supporting the prior theoretical background, suggests that the web – and in its substantial amount of information – is increasingly becoming a strategic variable for firms looking for new competitive advantages.

In addition, the web has influenced the habits, needs and behaviors of people, consumers and markets. Therefore, respondents highlight that firms are looking on the web for modalities to interpret the market's dynamics, sources of innovative intuitions and incentives for the development of dynamic capabilities to change their business models. Therefore, corporate strategies are increasingly relying on big data.

Table 2 summarizes the most representative quotes from interviews in relation to the research questions. Results are presented below with reference to the main elements of our research question.

4.1 How should dynamic capabilities be developed by firms to embrace digital innovation based on big data analytics?

Big data represent a strong growth engine for companies. Nowadays, companies have accumulated a huge amount of data… but they still do not know how to perfectly handle them. Properly interpreting data allows companies to know their potential customer, understand what he needs and how he will behave, and anticipate his purchase intent. We have to leave behind the concept of vanity metrics and focus on the information we can get by studying “what” and “when” derived from the data. The next step is to understand the related “why”: this is the biggest step to grow.

(Respondent 23)

Big data analytics push firms to change their innovation processes based on a logic in which the traditional boundaries between technology-driven and market-driven approaches disappear. Respondents' answers indicate that the potential of big data existed in generating information, which is often able to facilitate the development of innovation and the redefinition of firm-customer relationships.

Our executive informants consistently suggested that the potential to obtain information from big data was a critical advantage to strengthen “traditional” markets and corporate strategies. In this respect, the general view was that big data analytics facilitate companies' competitive repositioning and the creation of new advantages.

Then, how should firms capitalize on both the technology and the demand perspectives in the innovation process based on big data? Interviewees pointed out the potential to use big data to radically redefine the relationships between firms and customers, dematerializing a part of the supply chain. The web is the platform that enabled sales models to be redefined by “shops” and “new locations,” sometimes called “virtual” but more properly defined as “digital,” where supply meets demand by using technology as a tool for meeting, conversation and bargaining.

Indeed, the firm–customer relationship is changing; this relationship, as conducted over product sales processes, is becoming the starting point for product and manufacturing innovation. The traditional approach oriented toward understanding customers' needs and behaviors, including hidden ones, has also considered the potential of big data analytics. Moreover, the increasing competition and globalization of strategic processes are requiring a search for new positions of competitive advantage.

In current contexts, given the complexity of innovative phenomena, it is increasingly difficult, in fact, to frame innovative processes along a predefined linear path from a precise point of origin to one of arrival. Market-driven and technology-driven approaches are increasingly the extremes of a single path that, over time, has adopted countless intermediate and interactive forms in which the interweaving of the increasingly available scientific and technological knowledge with market dynamics is evident.

4.2 Which forms of dynamic capabilities should be developed by firms to embrace digital innovation from big data analytics?

Market and customer analyses, simulations and forecasts based on reliable statistics, scenarios oriented to support the most important strategic decisions, data management and selection according to the business, filing and rapid analysis of a limitless amount of information, management of unstructured data and interrelation with structured data, security and rapidity of the systems: these are the resources necessary to be competitive.

(Respondent 11)

Our findings suggest that firms should develop dynamic capabilities that are useful for effectively supporting innovation processes along multiple dimensions and orienting relationships with users/customers in new ways: in the “time”; in the “space”; in the “verso”; and in the “orientation”. Respondents also suggest the development of a dual approach to continuous and real-time interactions. The “time” for interactions is getting shorter, and firms need to develop new skills capable of promoting “real time” relationships with users and customers. The speed and agility of big data can lead firms to discover new capabilities and target marketing activities.

The challenge is to provide information within milliseconds to influence consumer choices in real time. In this sense, the frontier of big data analytics is real-time analytics solutions.

(Respondent 17)

Big data analytics also push a reconfiguration in the “space”. As the boundaries and the current distinctions between the traditional world and the “digital world” become more blurred, firms should develop new skills capable of analyzing these worlds overall.

Our results suggest an initial relevant insight: the web, with its different forms of evolution and different social networking platforms, has created a “real-traditional world” that is far from the idea of a “virtual world” suggested by many prior studies; In this “real-traditional world,” people are confronting, confiding in and sometimes lying to each other in a “real,” even different and digital, way. The people acting in this world act the same as in the “real world,” sometimes play a different role, and are the most important stakeholders for firms.

Second, with regard to the mechanisms that enable firms to leverage information management in corporate strategies for the “physical world,” our survey shows that the “digital world” complements and/or overlaps the “physical” world until they ultimately, sometimes, correspond or at least, without any doubt, influence each other reciprocally.

Third, firms' boundaries become wider, and they should develop new capabilities that can develop adequate links with users and customers. The strategic use of information technologies to redefine internal work relationships – teleworking – and interorganizational relationships – networks and virtual systems – offers new possibilities for innovation. In the digital world users/customers adopt increasingly real behaviors needing for effective collection and analysis of big data. Indeed, a relevant insight emerges with reference to the “verso”: there is a reciprocal influence that is not easy to understand but that drives both the researchers' and the executives' attention. By exploiting these two dimensions (digital and physical), firms can strengthen actions related to communication and marketing.

Today, we are witnessing an increasingly strong admixture of digital and traditional worlds. For example, in retail, we have increasingly integrated information flows. We try to know the customer – if properly registered on web browsers – even before he comes into the store, researching and obtaining on the web his preferences and spending characteristics, to be able to advise him more effectively. In the same sense, the purchases and behaviors of customers in the store are stored, recorded and organized to validate the research on the web.

(Respondent 6)

Additional insight is provided into the “orientation” of firm-user/customer relationships. The most innovative firms aim not only to understand and satisfy needs, hidden or not, but also to be active players in generating and changing both customer and market needs. In fact, during the interviews have emerged significant risks from the use of big data analytics. In particular, the first risk is that of “boring” users/customers. In this sense, firms should develop new capabilities that can shift the focus from traditional customer satisfaction towards the ability to generate a “surprise” effect. The second is related to the chance that mainstream forecasts and data will reduce firm creativity and decrease the development of high-risk radical innovations.

The customer will soon become narrow-minded in his tastes, becoming so bored, just because he receives always what he expects to […] In the future, firms should offer in the right way and successfully, the thing the customer considers, making a mistake, to be the wrong one.

(Respondent 3)

Overall, the integration of theoretical and empirical insights suggests that the ability of big data analytics to support the innovative strategies of firms has gained attention (Markides and Anderson, 2006; Lindstrom, 2016).

In the past, the main risk to face when using big data analytics was that the digital user would not exactly correspond to the customer in the “physical world.” In light of this potential disconnection, the use of information on the web to drive firm innovation should represent a risk for decision-making processes (Choi et al., 2017). This is why, in the past, the use of web data has been essentially confined to web strategies and web marketing by tools such as “guerrilla,” “cleaning” and “positioning” (Edelman, 2010).

However, our findings suggest that the time to open big data analytics to innovation processes in the “physical” world is very close. Big data analytics is useful for increasing the chances of understanding the habits, needs and behaviors of customers in the “real-traditional world” (Maglio and Lim, 2016). This change can drive firm innovation processes to rewrite the “rules of the game” (Davenport et al., 2012; Leeflang et al., 2014). Big data analytics are witnessing the development of innovative paths aimed at interpreting and integrating the two logics in a new way, aiming, from a more typically strategic view, to relaunch the “technology-push” logic alongside the accredited “demand-pull.” These two perspectives appear not antithetical but, conversely, should complement each other to increase their effectiveness. Big data analytics can drive innovation processes through the development of disrupting ideas that are able to model, convey and influence market demand, thus triggering a virtuous circular process, according to which big data analytics are, simultaneously, innovation achievements and the premise for further developments and new opportunities. The speed of big data analysis can overcome the dichotomy between sources of innovation and eliminates the possibility of delineating strict boundaries between technology and market needs. New capabilities are required to close the gap between the two worlds.

Firms should develop managerial activities that create a capability dynamic with regard to sensing by focusing on the new “space” of firm-user/customer relationships to identify and assess opportunities existing on the boundaries between worlds and between the inside and the outside of the firm (Teece, 2007). Big data analytics can play a key role in identifying and shaping opportunities at the crossroads between investment in research activity and learning about customer needs (Chen and Zhang, 2014; Jin et al., 2016).

At the same time, firms could mobilize their resources to capture value from new products, processes, or services by seizing on “timing” opportunities highlighted in big data analytics. Finally, a critical strategic element for capturing value from innovation is the ability of the innovating enterprise to identify and control the transformation and the continuous renewal needed for capabilities useful in the new “verso” and “orientation” of firm-user/customer relationships (Rapp et al., 2010; Bollinger, 2019). In the short term, a logic of customer satisfaction will survive. However, in addition to traditional logic, firms should develop a new logic of customer surprise that produces effects on relationships with users/customers in the medium term. Firms should orchestrate an attack on the threat of boring customers by introducing a new logic of customer surprise. Our results are summarized in Table 3.

5. Conclusions

This paper advances a comprehensive view aimed at supporting an innovation process based on big data analytics. The relations among digitization and innovation approaches have been investigated through the lens of dynamic capabilities and by an interpretive methodology based on interviews with leading experts at firms within digitally related sectors.

In this respect, our data show that the “digital world” and big data analytics are a new powerful source of innovation. The research also highlights an evolution in how companies embrace information and data coming from the “digital world” as components of corporate strategies and innovation processes. Investing in big data analytics can support innovation processes and provide new sources of innovation (Chen et al., 2012) to the extent that decision makers will have more information on market needs in the “digital” and “real-traditional worlds” and will view the “technology-push” and “demand-pull” perspectives in an integrative way (Dobusch and Kapeller, 2018; Bresciani et al., 2018). The results of this study highlight the development of a circular process between technology and customer needs that motivate firms to foster the evolution of innovation processes. Moreover, new dynamic capabilities are needed to exploit counterintuitive strategies aimed at developing innovative products and services.

We contribute to the literature on digitization and big data analytics. The “virtual/digital” space for negotiations and exchanges embraces new trends and actions with equal, or even higher, strategic content. The focus shifts from big data to rich data and/or deep data. With regard to rich data, firms should increasingly identify and select, within big data, information that can truly create value and contribute to decision-making processes by improving the strategies of firms (Levine et al., 2017). Regarding deep data, our study suggests overcoming traditional analytics trying to superficially expand and extend horizontal data (Lindstrom, 2016). Instead of “data surfing,” firms should identify crucial signals and then deep dive in these data. In this sense, the ability to identify and distinguish “sounds” from “noise” becomes a distinctive capability.

We also contribute to innovation studies. Originally, big data analytics provided the opportunity to identify the digital users' paths to propose similar solutions that would be even more appealing than those they found by themselves (Coussement et al., 2015; Jin et al., 2016; Peteraf et al., 2013). Later, big data analytics became able to find profiles similar to those of digital users to propose solutions that they had not considered before but that similar digital users had already looked for. The future appears to be also different and new for scholars and practitioners engaged in innovation issues (Dong et al., 2016). “Visionary managers of information” believe that firms at the forefront will soon be able to give to the customer exactly what he wants and expects. However, they think that to avoid boring customers, it will become truly and strategically relevant for firms to develop the capability to offer “in the right and successful way, the thing the customer considers, making a mistake, to be the wrong one.” This means offering customers something different from their expectations, assuming the risk of generating an apparent but temporary disappointment, and at the same time aiming to become for him a point of reference and the best guide for enlarging tastes and knowledge. Prior studies have highlighted the mutual importance of innovation sources (Di Stefano et al., 2012; Chaffey and Ellis-Chadwick, 2016). In addition to previous research, we suggest how big data analytics can favor the integration of the “technology-push” and “demand-pull” perspectives.

Moreover, we contribute to the stream of work on dynamic capabilities. Our study confirms the need for a continuous orchestration of capabilities by addressing the role of big data analytics and their positive impact on firms' innovation processes (Giudici et al., 2018; Teece, 2007). Specifically, digital transformation requires the evolution of capabilities along four main trajectories of firm-user/customer relationships: the “timing,” the “space,” the “verso” and the “orientation.”

Finally, this study also offers important practical implications. Our findings highlight how the current digital transformation is changing the way in which firms do business, with particular reference to innovation processes (Teece and Linden, 2017; Balusamy et al., 2017). This is probably the “future of the future” of innovation processes in the “physical world,” where “technology-push” and “demand-pull” perspectives complement each other as sources of firm innovation processes. The traditional distinction between the “technology-push” and “demand-pull” perspectives in firm innovation processes will become increasingly confusing and even “unreal.” Taking into account the difference between data bearing value (rich data) and data inspiring further verticalization (deep data), our work indicates the importance for firms to implement counterintuitive strategies aimed at developing innovative products, services, or solutions with characteristics that may diverge at first, even significantly, from established customer/user needs. Therefore, firms could benefit from investing in both in technology-driven and market-driven projects. Furthermore, new innovation processes may need new organizational models, new organizational roles and new methods of project management to support the orchestration of the required dynamic capabilities.

Our paper raises a number of questions for future research. Our results could be the starting point for future research on the relationships between big data, corporate strategies and innovation processes. It will be important to analyze case studies based on successful and unsuccessful experiences with the implementation of new technologies and tools; to analyze the impact of big data analytics-based innovation processes on customer satisfaction; and to advance extended frameworks that can integrate big data analytics, innovation perspectives and strategies for the physical and, maybe better, the “real-traditional,” world.

Table 1

Characteristics of interview respondents

Respondents	Organization	Expertise
1	Digital Agency	Web agent and certified Google partner Specialized in Virtual Reality projects, Strategic Communication and Mobile Apps
2	Full services sports Marketing Company	Co-founder Data scientist and digital consultant of important Football teams, Sport Federations and Media
3	Online Magazine and Database of Movie Information	CEO and co-founder Expert in managing information related to movies and cinema contents
4	Company of digital division of a leading publishing Group	Managing director of the “digital division” of Media Group Expert of Digital Media
5	Customer Journey Agency	CEO Specialized in digital marketing and web strategies
6	Digital Agency	President Specialized in digitization, knowledge management and innovation
7	Technology Company	General Director Specialized in Artificial intelligence, big data analytics and digital advertising
8	Global, diversified media, information and services company	Chief Digital Officer for Italy e Western Europe Specialized in digital content and Consumer Experience
9	“Business incubator” company for innovative start-up	Founder, Chairman and Chief technology officer Specialized in new digital technologies and selection of innovative projects and start-up
10	Company among the main rating agencies in Europe specialized in managing financial information	Head of Structured Finance, Investor Relations and ESG Specialized in Corporate Development
11	ICT company specialized for business drivers of change for its customers	Head of Innovation, Marketing and Technology Specialized in innovative technologies and in the development of an open innovation ecosystems
12	e-Commerce company that serves the technological evolution of Italian households	Chief Marketing Officer Specialized in market trends analysis and in product design and distribution format
13	Company in the field of cloud marketing technologies	Chief Innovation Officer Specialized in Data driven and Advanced Marketing Automation features
14	International e-commerce company specialized in engineering software solutions	Chief Executive Officer Specialized in internationalization, innovation, and profitable organic growth, accelerated via targeted acquisitions
15	International e-commerce company specialized in air navigation services	Innovation manager Specialized in digital transformation, ICT solutions and Enterprise Resource Planning
16	International e-commerce company specialized in air navigation services	CEO Specialized in analysis, planning and creation of projects for the computerized management of corporate procedures
17	Company in Big Data Analytics and Email Collaboration, supporting firms' Digital Transformation	CEO Specialized in big data management
18	Sports digital contents company	Managing Director Specialized in Media Content
19	Web services company	Managing director Specialized in managing Multination IT Companies
20	Multimedia production company with creative and experimental approach	Founder Specialized in movie direction, movie production and cross-media
21	Multimedia publishing company	CEO Expert of Digital Media
22	Innovative start-up in IoT and Industry 4.0 solutions	CEO Expert of IoT and technological innovation
23	Leading Data-driven company in Europe	Head of Strategy Specialized in strategic management
24	Consulting firm, specialized in systems integration and technology	Business Analytic and Consultant Specialized in strategic and business intelligence
25	Tour operator, specialized in “experiential” tourism	Co-founder Expert of digital communication

Table 2

The most representative quotes from interviews

	Research questions	Representative interview quotes
1	Which are the required capabilities to embrace a successful digital transformation based on big data analytics?	“To have a cutting-edge technology able to process big data – recognizing rich data and deep data, through models and algorithms, which are foundations of machine learning analysis – more and more represents an element of success for firms.” Respondent 4
		“Increasingly, the behaviors of users/customers in the digital world are more real than we could observe in everyday life; for these reasons, it is essential that big data analytics are structured to maximize the efficiency of the collection, classification, analysis and synthesis of web data.” Respondent 18
		“The user is increasingly aware, informed and interactive in his relationship with brands: he is always on and demands that his wishes and questions are answered at the exact moment he needs. We must therefore be able to know the user in depth; this will allow us to adapt our product or service around his real needs in a specific and very short time span. The added value is to intercept him in the micro-moment in which he is ready to listen to us.” Respondent 14
2	Which capabilities are critical in the development of innovation processes?	“We could have an idea in the morning and then be developing it in the afternoon if need be. We've had things where we've turned around a full solution in 24 hours. This was unheard of before.” Respondent 19
		“Perhaps a risk of using big data for innovation could be to be overly influenced by strong forecasts and reliable scenarios, but still outlined and influenced by the main-stream (how-ever complex), barring the way for more reckless creative jumps that are instinctive and (at least apparently) irrational (which in the field of innovation can sometimes be effective)?” Respondent 20
		“In the current economic context, it is important for firms to promote the creation of a collaborative business model to identify new opportunities among the value chains of the firm and the value chains of external stakeholders; it is a complex process that determines changes in the way of competing by modifying the relationship between the firm and its external environment. Subsequently, big data analytics can not only support the integration processes, but mostly, starting from these interactions, big data can truly release their potential by increasing firm's capabilities for data acquisition and interpretation. In this way, decision-making processes are developed based on several pieces of information provided out-side the firm's boundaries, allowing us to more correctly define objectives and goals.” Respondent 16
3	How and to what extent is the digital transformation changing the innovative processes of firms, in terms of new products and services?	“The ‘cross-reference’ activities are already advanced and operative; these activities are not only able to advance, in real time, preferences that are similar to those the digital user already searched on the web but also to suggest things that users with a similar behavioral profile have searched.” Respondent 7
		“Thanks to big data, firms can have access to much important information about their business, but the main challenge is to be able to create new value from it. We have developed several solutions to successfully manage big data. For example, a cloud platform that analyses and studies customer behavior thanks to a multichannel system. It combines online channels (newsletters, apps, social networks, communities) and physical elements present in the store (interactive screens, sensors, smartphones, tablets) to send personalized communications to every single person, improving contact and the company–customer relationship, both online and while in the store.” Respondent 9
		“The future of big data concerns the ability to intercept the customers in the right place and at the right time with increasingly personalized and relevant communications.” Respondent 10
		“The technological platform suggests new products, promotions and dedicated messages based on customer profiling. When a person enters the store, these tools can indicate if it is a customer to be rewarded, motivated, or retained and whether that person is an occasional or frequent shopper.” Respondent 9
		“It is important to recognize the value of platforms designed for brands, retailers, producers and designers, with the aim of customizing and personalizing products, thanks to ‘virtual re-tail and 3D commerce’. These platforms enable customers to see the 3D simulation of clothing and footwear worn to personalize and purchase them. It turns stores into new ‘phydigital’ conceptual spaces, offering an innovative virtual shopping experience.” Respondent 5

Table 3

The “big data analytics”-based evolution of dynamic capabilities for innovation processes

Firms-users/customers Relationships	The past	The future	Implications for dynamic capabilities
“Timing”	There was traditionally a lag between the time for technology development/customer's needs perception and the time for customer satisfaction	There will be a dual approach with real time interactions between firms and user/customers	Need for capabilities useful to increase the speed and the agility alongside the foresight necessary to pursue a strategic duality
“Space”	There were two separate worlds: the traditional/physical world and the “Digital/virtual” worlds	There will be a single world where the distinction between traditional and digital disappears and where the people act	Need for capabilities useful to the management of boundaries (between worlds; between inside and outside)
“Verso”	There were influences from the companies to the customers or from the customers to the companies	There will be mutual, reciprocal and relevant influences between, from a side, firms and companies and, on the other side, physical and digital worlds	Need for capabilities useful to rewrite the “rules of the game” increasing the chances of understanding habits, needs and behaviors of customers
“Orientation”	There was a traditional logic of customer satisfaction	Besides a logic of customer satisfaction in the short term, there will be a new logic of customer surprise with effects in the medium term	The capabilities for identifying and distinguishing “sounds” from “noises” and “foregone” from “amazing” becomes distinctive