Riding a bicycle while building its wheels: the process of machine learning-based capability development and IT-business alignment practices

Abstract

Purpose

Recent advancements in Artificial Intelligence (AI) and, at its core, Machine Learning (ML) offer opportunities for organizations to develop new or enhance existing capabilities. Despite the endless possibilities, organizations face operational challenges in harvesting the value of ML-based capabilities (MLbC), and current research has yet to explicate these challenges and theorize their remedies. To bridge the gap, this study explored the current practices to propose a systematic way of orchestrating MLbC development, which is an extension of ongoing digitalization of organizations.

Design/methodology/approach

Data were collected from Finland's Artificial Intelligence Accelerator (FAIA) and complemented by follow-up interviews with experts outside FAIA in Europe, China and the United States over four years. Data were analyzed through open coding, thematic analysis and cross-comparison to develop a comprehensive understanding of the MLbC development process.

Findings

The analysis identified the main components of MLbC development, its three phases (development, release and operation) and two major MLbC development challenges: Temporal Complexity and Context Sensitivity. The study then introduced Fostering Temporal Congruence and Cultivating Organizational Meta-learning as strategic practices addressing these challenges.

Originality/value

This study offers a better theoretical explanation for the MLbC development process beyond MLOps (Machine Learning Operations) and its hindrances. It also proposes a practical way to align ML-based applications with business needs while accounting for their structural limitations. Beyond the MLbC context, this study offers a strategic framework that can be adapted for different cases of digital transformation that include automation and augmentation of work.

Citation

Mucha, T., Ma, S. and Abhari, K. (2023), "Riding a bicycle while building its wheels: the process of machine learning-based capability development and IT-business alignment practices", Internet Research, Vol. 33 No. 7, pp. 168-205. https://doi.org/10.1108/INTR-10-2022-0769

1. Introduction

The rapid progress of machine learning (ML) advancements is at the core of the current wave of artificial intelligence (AI) commercialization (Berente et al., 2021). A number of widely publicized ML use success cases and the resulting value creation in companies such as Alibaba (Wang et al., 2018; Zhang et al., 2021b), Google (Knight, 2018) and eBay (Brynjolfsson et al., 2019) have fueled corporate investment in ML-based digitalization initiatives around the world (Zhang et al., 2021a). As an emerging type of general-purpose technology (Goldfarb et al., 2019), ML technology is having increasingly significant impacts on organizational value creation, business models and competitiveness (Ransbotham et al., 2019), and many business leaders understand that ML is highly important and disruptive digital technology (Benbya et al., 2020). However, despite the relative ease of piloting ML projects in organizations, scaling and deploying them have proven to be challenging (Benbya et al., 2020; Lwakatare et al., 2020b; Wang et al., 2018; Zhang et al., 2021b). Indeed, only one in 10 companies implementing ML initiatives have realized satisfactory outcomes (Ransbotham et al., 2020). The mismatch between the newly unlocked raw technical capabilities of ML and the productive harnessing of organizational ML-based capabilities (MLbC) has been called the “modern productivity paradox” (Brynjolfsson et al., 2018).

MLbC refers to the organizational ability to align ML-specific and other resources to perform a particular activity in a reliable, repeatable and value-added manner. The emphasis on alignment is rooted in past research on organizational and IT capabilities within the resource-based view (RBV) literature (Amit and Schoemaker, 1993; Bharadwaj, 2000; Helfat and Peteraf, 2003; Sirmon et al., 2007). Prior studies have shown that the mechanism for capability development can be regarded as an alignment process. This perspective is also consistent with the IT-business alignment literature (Chan and Reich, 2007; Luftman et al., 2017). It is therefore reasonable to expect that capability development generally unfolds as an iterative and continuous process of alignment (Helfat and Peteraf, 2003; Luftman et al., 2017), thus necessitating the use of a temporal lens to study the alignment between organizational resources (Ancona et al., 2001). Additionally, to attain insight into the alignment process, we must consider both technological and organizational and individual aspects of alignment since organizational resources reside at different levels (Lyytinen and Newman, 2008; Sirmon et al., 2011) and include both operant and operand resources (Bharadwaj, 2000). Likewise, the alignment process is sensitive to changes in the organizational context and external environment (Lyytinen and Newman, 2008; Sirmon et al., 2007). Although these three assumptions are well-accepted, the emerging literature related to MLbC has thus far failed to account for the organizational aspects of MLbC development – in particular, its temporal and contextual dimensions. Two complementary streams of literature—technology-focused research on ML operations (MLOps) (Choudhary et al., 2022; Mäkinen et al., 2021; Renggli et al., 2021) and organizational research on AI/ML use and management (Berente et al., 2021; Mikalef and Gupta, 2021)—also lack an overarching framework that systematically accounts for the MLbC development process. These two interrelated theoretical gaps motivate our research question, which includes practical implications: What mechanisms facilitate the development of machine learning-based capabilities?

To answer this question, we conducted an in-depth exploratory study to integrate the disjoined views on the MLbC development process. The context of our study is companies affiliated with Finland's Artificial Intelligence Accelerator. The participating firms included some of the largest organizations in Nordic countries representing a variety of industries, including telecommunications, banking, retail and media broadcasting. To complement this extensive data and allow for triangulation, we conducted additional confirmatory interviews with organizations outside the accelerator based in Europe, China and the United States. Overall, we analyzed 149 data events with an approximate total length of 175 h over four years. Analysis of this rich and nuanced dataset allowed us to theorize MLbC development processes and contribute to the literature in three notable ways.

First, we bridge the gap between MLOps and information systems (IS) literature by articulating the missing organizational process of MLbC development, which thus distinguishes our research from the ML-based application development portrayed by MLOps literature. Our examination of this overlooked area revealed the importance of the temporal and cyclical aspects of the alignment process constituting MLbC development. Specifically, we highlight that MLbC development iteratively progresses through (re)initiating, effectuating and sustaining alignment phases and show that the interaction of MLbC's structural components and their respective alignment practices governs this progression. Second, we contribute to the emerging literature on AI and ML technology management. We facilitate meaningful IS conversation on MLbC development by offering contextually rich and fine-grained insight and explaining the mechanisms that govern the process. Third, we contribute to RBV and IT-business alignment literature, both of which emerged at a time when instructions had to be explicitly codified for the technology to work. Nevertheless, contemporary organizations are increasingly looking for opportunities to automate and augment their processes with minimum human interventions (Ågerfalk, 2020; Berente et al., 2021; Lyytinen et al., 2020). This challenges some long-held beliefs about technology being a relatively fixed asset with a limited intentionality (Chan and Reich, 2007). Therefore, our study contributes insights into this novel area residing at the intersection of organizational capabilities, IT-business alignment and non-deterministic technologies.

2. Research background

Organizational capabilities are the key to continuous innovation and sustainable competitiveness. The literature about the RBV (Barney, 1991; Wernerfelt, 1984) and resource orchestration (RO) (Sirmon et al., 2007, 2011) define organizational capabilities as an organization's ability to align the deployment of internal resources and competencies to achieve optimal end results (Amit and Schoemaker, 1993; Bharadwaj, 2000). Prior research specified that organizational capabilities could not be easily bought; thus, they are often developed internally in alignment with organizational goals (Sirmon et al., 2007). Organizational capabilities and dynamism must also be orchestrated and maintained to derive results (Sirmon et al., 2011). Therefore, a capability represents more than a one-off effort to perform a task or set of tasks in an organization; instead, “for something to qualify as a capability, it must work in a reliable manner [… and it] must have reached some threshold level of practiced or routine activity” (Helfat and Peteraf, 2003, p. 999). To reach this threshold, creating, utilizing and maintaining a new organizational capability requires reasonable alignment between the key resources—people, processes and technology—to create that capability (Amit and Schoemaker, 1993; Helfat and Peteraf, 2003). Alignment in this context refers to “both doing the right things (effectiveness), and doing things right (efficiency)” (Luftman et al., 1999, p. 4). A desirable level of alignment can only be achieved over time with proper coordination and integration, whose function is even more vital when a new capability is formed by building on or extending existing capabilities (Bharadwaj, 2000; Sirmon et al., 2007), or when the new capability disrupts work practices and introduces new routines (Ancona et al., 2001; Ulrich and Lake, 1991).

New capability development can be characterized as an iterative, multi-level and context-sensitive process. First, it is an iterative process that involves multiple trials, searches for alternatives and ambiguous feedback (Helfat and Peteraf, 2003). Temporal progression is, thus, a fundamental characteristic describing the underlying processes, including the alignment process (Ancona et al., 2001). Second, organizational capabilities reside at different levels of an organization (Sirmon et al., 2011), and lower-level capabilities combine into higher-level specialized capabilities (Grant, 2016). This means that the alignment between organizational routines and the efforts to develop capabilities at different organizational levels must be accounted for in the process and governance (Grant, 2016). Additionally, individuals, along with their roles, cognitive representations and organizational setting affect the process and outcome of the capability development (Barney and Felin, 2013; Gavetti, 2005). Therefore, we must recognize the role that individuals and organizational structures play in capability development and the coordination required to align their contributions. Third, capability development is both constrained by and geared toward achieving alignment with the internal and external environmental context (Gavetti, 2005; Sirmon et al., 2007). Dynamic response to internal and external opportunities and challenges is also necessary to maintain alignment between the constituents of a capability throughout its development process. Hence, constant monitoring of the environment (e.g. sensing) and responding to changes within the environment (e.g. reconfiguration) are necessary to maintain alignment between a new capability development process and its environment (Wilden et al., 2016).

Developing technological capabilities is also iterative, multi-level and context-sensitive; therefore, its success depends on proper alignment between its constituents. For this reason, IT-business alignment has been and continues to be a central topic of interest for IS scholars (Chan and Reich, 2007; Luftman et al., 1999, 2017; Trang et al., 2022) and referred to by a variety of terms, such as fit, fusion, integration, linkage or harmony (Chan and Reich, 2007; Luftman et al., 2017). The unifying theme of IT-business alignment literature is its focus on “coordinating activities across IT and non-IT domains within the firm” to provide new organizational capabilities (Luftman et al., 2017, p. 27). This representation of IT-business alignment in IS literature, as a necessity for organizational capability development, indicates intertextual coherence (Locke and Golden-Biddle, 1997) with RBV and RO literature and solidifies the conceptual grounding of our work in three ways.

First, IT-business alignment is an ongoing process, not necessarily an outcome (Luftman et al., 2017). It is widely acknowledged that IT-business alignment requires continuous efforts (Chan and Reich, 2007; Lyytinen and Newman, 2008), thus emphasizing the need for a temporal lens in studying IT-enabled capabilities (Chan and Reich, 2007). Second, IT-business alignment is stratified into different levels (Chan and Reich, 2007; Lyytinen and Newman, 2008), ranging from inter-organizational (Trang et al., 2022) and organizational (Henderson and Venkatraman, 1999), down to individual alignment with IT (Goodhue and Thompson, 1995). Third, as a consequence of this unfolding on different levels, it is also recognized that there are interrelationships between the levels (Chan and Reich, 2007; Lyytinen and Newman, 2008). Thus, the success of new capability development depends on IT-business alignment, which is sensitive to events in the external environment and equally to the internal organizational context (Lyytinen and Newman, 2008).

These three assumptions suggest that developing MLbC capabilities and aligning them with business goals and strategies cannot be successfully achieved with episodic development, isolated from organizational structure and business processes, and independent from the business context. Although the IT capability literature traditionally recognizes the importance of these three assumptions at macro and meso levels (Bharadwaj, 2000; Mikalef et al., 2020; Mikalef and Gupta, 2021), AI and ML management research falls short in identifying what should be aligned, much less how to achieve this alignment. Building on this gap, we draw on related work in MLbC development and management, where the focus is on how time and context play a role through alignment at an operational level.

2.1 Machine learning-based capability development in organizations

Based on the key aspects of organizational capability development through alignment, we distinguished two dominant perspectives assumed by past research into MLbC development. These perspectives are ML Operations and Organizational Use of ML. Table 1 summarizes how these two perspectives address key aspects of MLbC development in a review of the literature on capability development and IT-business alignment research.

The key limitation of the MLOps perspective is its inadequate account of the organizational aspects in the development of MLbC, and the Organizational Use of ML literature's main constraint is an insufficient understanding of MLbC's temporal aspects. Both perspectives fail to include the contextual detail and nuance required to capture the richness of MLbC development in organizations. In the following sections, we provide an overview of these perspectives and elaborate on their limitations as relevant to advancing the theorization of MLbC development.

2.1.1 MLOps perspective

MLOps is a nascent yet rapidly growing field of research and practice. Figure 1 presents the relative popularity of the phrase “MLOps Topic” in Google Search and the number of academic papers with “MLOps” in the title, abstract or keywords.

MLOps has grown out of Development Operations (DevOps) frameworks. DevOps has been popularized for developing and operating modern IS in an integrated fashion (Alnafessah et al., 2021; Gall and Pigni, 2022; Moreschini et al., 2022), and its framework includes two intertwined cycles: a development cycle (planning, coding, building and testing) and an operation cycle (release, deploy, operate and monitor). DevOps's goal is to release new software features quickly and without interruption to the software operation through rapid cycles of frequent changes, continuous integration and continuous delivery (Mäkinen et al., 2021). Therefore, DevOps is not considered a plug-and-play solution (Mäkinen et al., 2021), but a useful framework for addressing practical challenges in the increased organizational implementation of ML technologies. MLOps emerged as a new approach to DevOps specifically geared toward ML technologies (Alla and Adari, 2021; John et al., 2021; Mäkinen et al., 2021; Moreschini et al., 2022).

MLOps is a specialized set of practices that aim to address the unique challenges of implementing Machine Learning technologies in production. These challenges include aspects such as model construction, training and monitoring, as well as high requirements for data workflows, such as data cleaning, analysis, validation and feature extraction. Additionally, in an MLOps environment, there are complex dependencies and feedback loops between code, model and data artifacts that must be carefully managed to ensure optimal performance and reliability (Amershi et al., 2019; Mäkinen et al., 2021; Renggli et al., 2021; Zhou et al., 2020). The latest MLOps literature focuses on the process and timing of operations, as well as technical aspects. This, however, results in simplifying the problem of MLOps' sensitivity to external context and viewing it primarily through data.

First, MLOps is viewed as a process and can be characterized as either a DevOps cycle with ML components (Moreschini et al., 2022) or a DevOps cycle dedicated to ML technology development (Lwakatare et al., 2020a; Martínez-Fernández et al., 2021). Although scholars propose multiple alternatives for integrating ML components (e.g. model development, data management) with DevOps' development and operations cycles, the consensus is that MLOps is a cyclical process with multiple feedback loops over time (Lwakatare et al., 2020a; Mäkinen et al., 2021; Paterson et al., 2021; Renggli et al., 2021). Despite embracing this process viewpoint, MLOps studies primarily focused on the temporal factors of technology development (e.g. updating models in time) and operation (e.g. collecting feedback in time) at an application-level (Huang et al., 2022) and overlook temporality such as timing, pace and synchronization with broader organization and users (Muralidhar et al., 2021). Recognizing this gap, we follow a recommendation from Choudhary et al. (2022) and distinguish development, release and operation as three key phases in the MLOps cycle to capture the temporality of MLbC development with more granularity.

Second, research showed that operationalizing MLOps at scale requires deep integration between development and organizational processes (Mäkinen et al., 2021) that is not independent of aligning the ML development and operation with the organization's goals and context. However, MLOps frameworks do not sufficiently account for this necessary alignment (e.g. possible interactions between human and non-human agents to realize the full potential of these solutions). Current MLOps models are also limited in explaining human–machine dynamics, from delegation to negotiation. Recent studies argued that the MLOps cycle, unlike typical DevOps, cannot be simply integrated with organizations for various reasons, from the absence of a systematic ML pipeline (Moreschini et al., 2022) with context-specific solutions (Garcia et al., 2018) to the lack of knowledge on governing ML-based solutions (Lwakatare et al., 2020a) and delegation mechanisms (Baird and Maruping, 2021). Moreover, there are some sociotechnical challenges concerning the MLOps-business alignment, such as establishing users' trust, increasing explainability, harnessing controllability and facilitating self-adaptation (Abedin, 2022; Laato et al., 2022; Martínez-Fernández et al., 2021), mainly due to limited empirical investigations into the human aspects of managing ML-based solutions in organizations (Asatiani et al., 2020, 2021; Keding and Meissner, 2021).

Finally, MLOps frameworks assume that solution adaptation to operational context changes can be simplified through data processing and automated model re-training (Choudhary et al., 2022; Paterson et al., 2021), allowing organizations to actively adjust operations to meet constant fluctuations in external or internal environments. For example, Google lowered its power consumption in data centers by handing over control of data centers' cooling to an ML algorithm (Knight, 2018). Similarly, a study of an ML-based digital twin set in the context of a petrochemical plant reported an improved production yield (Min et al., 2019). However, post-implementation, MLOps must be carefully monitored by human actors (Lyytinen et al., 2020; Symeonidis et al., 2022) due to their error-prone nature (Asatiani et al., 2021). This observation and, when necessary, intervention require significant coordination and operational employees' involvement. These aspects of MLOps-business alignment go beyond the scope of most MLOps practices in today's organizations (Pääkkönen et al., 2020).

2.1.2 Organizational use of ML

Spurred by organizations' rapid adoption of ML and the resulting challenges (Benbya et al., 2020; Ransbotham et al., 2021), both IS and organizational scholars increasingly have turned their attention to MLbC (Benbya et al., 2021; Mikalef and Gupta, 2021), their development (Zhang et al., 2021b) and their management (Raisch and Krakowski, 2021) to complement the technology-oriented perspective, MLOps. Despite the growing body of literature on the Organizational Use of ML, there are still several areas that require more attention. For instance, while some research has been conducted in this area, there is a need for greater recognition of the importance of the temporal aspects and the process of ML-based decision-making. Additionally, existing studies tend to focus on the role of ML users and decision-makers, often abstracting away from technological details, which can obscure crucial factors influencing the success of ML implementation. Furthermore, it is important to recognize that internal organizational context plays a significant role in shaping the development and effectiveness of ML-based decision-making processes. First, ML/AI management literature rarely uses a temporal lens to investigate MLbC development outside core ML technology development. For example, Mikalef and Gupta (2021) have drawn on RBV and identified the organizational resources needed to develop MLbC. Their work extends the perspective beyond the technical aspects, yet provides only a static view of MLbC's building blocks, leaving process and temporal aspects for future investigation (Mikalef and Gupta, 2021). A study of ML used in Alibaba's warehouse provides further insights into how technology and human resources can be bundled together to complement each other's strengths (Zhang et al., 2021b). However, the process's temporal progression and feedback loops are missing, despite recognizing these aspects' importance (Benbya et al., 2021; Mucha et al., 2022; Raisch and Krakowski, 2021; Sturm et al., 2021a). A growing number of process-oriented qualitative studies exploring ML use in organizations counterbalances some of these shortcomings. For example, the importance of multiple iterations and feedback loops, as well as the complexities and uncertainties related to timing, pace and rhythm of developments, have been recognized in recent studies (van den Broek et al., 2019; Grønsund and Aanestad, 2020; Mucha et al., 2023a; Ruissalo et al., 2022; Sturm et al., 2021b; Waardenburg et al., 2022). These studies, however, often concentrate on the development and release phases, lacking insights into the operational phase.

Second, this research stream has emphasized the human roles and structures within organizations as important drivers of ML development, release and operation. Thus, in line with the view that operand and operant resources are jointly needed to develop MLbC (Mikalef and Gupta, 2021), we should recognize ML users and representatives of technology development teams as key actors in the process. In a study of ML use by the consumer lending unit of a German bank (Strich et al., 2021), loan consultants not only relied on an ML-based application but also developed ways of working around it. We also recognize the emergence of new managerial and operational roles following the implementation of MLbC (Berente et al., 2021; Grønsund and Aanestad, 2020; Strich et al., 2021; Waardenburg et al., 2022) that frequently catered to unique requirements imposed by data collection and ML model (re)training and shaped organizational use and interpretation of ML-model outputs. These findings enrich our understanding of how users and data scientists might shape organizational MLbC. However, the nascent stage of this research still leaves many questions in our understanding of multiple roles and organizational levels' impact on MLbC. We particularly recognize the limited attention on the changes within the team or organizational sub-unit developing ML technology and their influence on MLbC (Fountaine et al., 2019).

Finally, organizational and IS research on the role of context in MLbC development is only starting to emerge (Mucha et al., 2023a). Conceptual work recognizes the importance of context changes, but there needs to be an agreement regarding the consequences of these changes. Balasubramanian et al. (2022) argue that augmenting human decision-making with ML is likely to be more beneficial when the external context changes at a high rate or magnitude. On the contrary, results from a simulation study by Sturm et al. (2021a) indicate that in volatile environments, the role of people is essential in searching for new knowledge and actively (re)training ML models to renew MLbC. Empirical studies also recognize the sensitivity of MLbC development to external contexts such as fluctuations in consumer buying patterns (Zhang et al., 2021b), the prices of raw materials (Min et al., 2019), customer profiles (Mucha et al., 2023a) and stock market trends (Sturm et al., 2021b). However, these studies primarily rely on relatively short observation windows, thus giving only preliminary results.

In summary, to overcome the operational limitations of MLOps and MLOps-business alignment, the notion of MLOps should be revisited from an organizational perspective for mainly three reasons. Firstly, current literature theorizations fall short in considering the temporality of MLbC development as an alignment process occurring throughout the phases of development, release and operation (van den Broek et al., 2019; Grønsund and Aanestad, 2020; Strich et al., 2021). Secondly, the operation of ML-based solutions requires organizational commitment across multiple levels and functions (Raisch and Krakowski, 2021). Lastly, the development of MLbC is not independent of the organizational context (structure and stakeholders) since the outcome of MLOps can radically change how the stakeholders perform different tasks and, in turn, how the organization fulfills its operational goals within its structure (Asatiani et al., 2021). These limitations emphasize the importance of an overarching framework that systematically accounts for the technical, organizational, behavioral and temporal aspects of ML-based solutions. Advancing our understanding of MLbC development and alignment, we study how companies orchestrate them over time.

3. Method

3.1 Study settings

Our primary study setting was Finland's Artificial Intelligence Accelerator (FAIA). FAIA's mission and activities revolved around the development of MLbC within its participating organizations, which allowed us to collect theoretically reliable and useful data (Eisenhardt, 1989). FAIA was initiated and originally funded by the Finnish government in 2018 to facilitate collaborations related to ML, stimulate adoption of the technology within the participating organizations, extract key lessons, as well as inform a broader audience of IT and business leaders in Finland on ML development, release and operation. Unlike start-up accelerators, FAIA focused its efforts primarily on established organizations. Firms participating in FAIA included some of the largest Nordic companies, such as Elisa (telecom operator), Nordea (bank), Posti (Finnish national postal services), S-group (retail chain), Telia (telecom operator) and YLE (Finnish national broadcasting company). Furthermore, multiple startups participated in FAIA in the role of ML service providers. Thus, the participating organizations provided a diverse sample of organizations investing in ML across various industries in Finland and with different levels of ML maturity. FAIA participants formed several semi-formal groups (batches), typically containing 4 to 8 members. Each batch focused on specific types of ML-based solutions or ML-related practices. The batches met regularly throughout the acceleration period for approximately six months. FAIA's team facilitated and catalyzed collaboration by creating a community of practice with peer support and peer pressure.

To complement our fieldwork centered on FAIA, we further engaged with organizations that were not affiliated with FAIA. These organizations included large corporations, consulting firms and research institutions based in Europe, China and the United States. We conducted interviews with employees in roles related to ML development, release and operation at these non-FAIA affiliated organizations. Data from these interviews allowed us to triangulate our analysis (Eisenhardt, 1989).

3.2 Data collection

As presented in Table 2, most of our data originated from the primary study setting, for which the data collection covered the first 3.5 years of FAIA's activities (August 2018 to December 2021). The first author engaged in observation and carried out the field study by participating in FAIA's weekly internal team meetings and workshops with the accelerator's participating companies. Interviews with the participants and other experts were also utilized when collecting data within and outside of FAIA. The first author also had unrestricted access to FAIA's internal documentation and was copied on part of the emails between the FAIA team and the participating companies. To the extent possible, the meetings, workshops and interviews were recorded and transcribed (alternatively, notes were taken during or immediately after interactions with the participants). Most of the events were in English, but some were in Finnish or Chinese (transcribed and translated into English). Overall, our data covers 149 data collection events with an approximate total length of 175 h. As a screening mechanism during the interviews, we first discussed the informants' roles in their organizations and their involvement in ML-related activities. Our questions concentrated on the actors, technologies, tasks and organizational structures involved in these activities. We also examined organizational practices and approaches, as well as encountered challenges and subsequent responses related to ML initiatives. When necessary, we asked follow-up questions and sought further clarifications. Appendix provides a summary of the informants' profiles. During our data collection, we focused on developing insights into ML-related activities, events their relationships, and their contribution to the development, release and operation phases of MLbC. We also stayed up to date on the developments in MLOps and general topics related to managing ML initiatives in organizations by actively reading articles and listening to interviews geared toward practitioners.

3.3 Data analysis

To analyze the data (Figure 2), we relied on qualitative case study research processes (Eisenhardt, 1989; Yin, 2009). We started analysis during the data collection process to guide our discussions and interviews with the participants and to continue elaborating on finer details of the organizational process for the development of MLbC. Simultaneously, we engaged with multiple streams of literature that guided us in constructing the preliminary framings for the research problem and refining the research framework (Sarker et al., 2013). Furthermore, to deeply immerse ourselves in the topic, we followed practitioner publications and podcast interviews relevant to MLOps and ML use in organizations. This approach allowed us to practice engaged scholarship (Van de Ven, 2007), which is particularly important for developing insight into the social and technological complexities of organizational phenomena related to ML technologies (Lyytinen et al., 2020).

We analyzed the collected data in three phases—open coding, thematic analysis and cross-comparison—to go beyond initial impressions and assume multiple perspectives on the evidence (Eisenhardt, 1989). To conduct a thorough analysis of the data, an open coding approach was implemented. Our analysis focused on three primary components: the significant milestones associated with the development of MLbC, the challenges faced during this process and the various coping mechanisms utilized by the participating organizations to tackle these challenges.

The first coding round resulted in a comprehensive understanding of MLbC development and the importance of MLbC-business alignment. Two major themes also emerged as part of the MLbC development challenges in tandem with the coping mechanisms: time and context. Based on our observation that participants frequently referred to the importance of time and iterations, as well as insights from theoretical and practical sources, we concluded that considering temporal progression and the cyclicality of the MLbC development process was of paramount importance. We also observed that MLbC development challenges were also functions of the MLbC context, namely task context, organizational context and business environment. Therefore, the second data analysis phase clarified the relationships between codes concerning the MLOps phases (development, release and operation) and MLbC development challenges across time and context. This allowed us to group the codes and identify second-order themes. In the third phase, we carried out additional analysis of the individual organizations participating in FAIA and cross-case comparisons to identify aggregated dimensions and core categories of concepts in our theorization. To further triangulate our results and minimize the risk of sample selection bias, we compared our findings with the insights gathered from non-FAIA-related sources. We found that the results from the two data sources were consistent. No new themes emerged from this analysis, thus indicating theoretical saturation. The triangulation process (Eisenhardt, 1989) also allowed us to develop a common language and a more universally accepted vocabulary to explain the MLbC development phenomenon. To ultimately reach our findings, presented in Figures 3–5 and expanded on in the next section, as well as the resulting theorization, we arranged multiple workshops where we iteratively moved between empirical data, emerging theoretical abstractions and background literature. Furthermore, we actively worked on and communicated our insights through visual representations to supplement our analysis, clarify thoughts and enhance the write-up process (Saldaña, 2015). In documenting the findings, we attempted to offer theoretical abstractions, as well as position and discuss our findings in the context of recent theoretical developments in the field.

4. Findings: development of machine learning-based capabilities through alignment

The analysis of codes and themes indicated that successful MLbC development critically depends on a broader range of factors than technology alone. The following quote aptly presents this viewpoint.

The fact that there is AI in the middle of [solution name] is almost unimportant. The point is now [users/customers] they have got a capability they did not have before, which we have made work by understanding the technicalities of AI, the way that the AI works, all the complexities of actually embedding it in the system, getting the human in the loop, putting in front of the customer in a way that makes sense to the customer. ∼Digital Engineering Lead

Therefore, our analysis centered on the alignment of MLbC development, release and operation at an organizational level. The capabilities based on ML, which emerged in the organizations we studied, took the form of digital artifacts turned into digital capabilities by virtue of interactions between three interrelated cycles. We labeled these cycles: ML Organization Cycle, ML Technology Cycle and ML User Cycle. We define alignment as consistent and compatible interaction between the three cycles, thus engendering the enhancement or creation of digital capabilities. Interactions of these cycles did not, however, automatically lead to alignment. On the contrary, many organizations faced challenges on the way. Our analysis revealed two important types of structural characteristics of MLbC that inhibited alignment efforts, namely Temporal Complexity and Context Sensitivity. In response to these challenges, successful organizations employed two sets of alignment practices—Fostering Temporal Congruence and Cultivating Organizational Meta-learning.

We report our findings in more detail in the subsections below. We first discuss how the organizations we studied strategically or inevitably move beyond MLOps to develop and integrate MLbC. Then, we report on the challenges they faced when aligning the constituent cycles of MLbC. Lastly, we conclude with two practices that emerged as the best to address these challenges.

4.1 Beyond MLOps: three constituent cycles of a machine learning-based solution

The MLbC development, release and operation process included more than a cyclical progression of technology through these phases, as is commonly depicted in MLOps literature. We define the ML Technology Cycle as a series of successive states, configurations and processes pertaining to artifacts, including data, ML models, infrastructure and software code related to MLbC development, release and operation. In addition to ML Technology, we have also identified ML Organization and ML User cycles. The ML Organization Cycle is a series of successive states, structures, and functional roles pertaining to a temporary or permanent social subsystem formed under the umbrella of the focal organization and which controls resources, enacts processes, sets rules and objectives, and possesses key competencies related to MLbC development, release and operation. By ML User Cycle, we mean a series of successive states and functional roles of organizational or external human agents who directly interact with or benefit from MLbC.

In the MLbC development phase, the three cycles are designed for and directed toward alignment. However, at that phase, alignment is only (re)initiated because the envisioned solution or its future iteration is yet to be implemented. In the release phase, the constituent cycles meet each other, thus putting the MLbC alignment effectively into practice. In the operation phase, alignment needs to be continuously achieved and sustained (Figure 6). If this is not the case, the process needs to iterate again or rewind to an earlier phase. The findings that led us to the theorization of the constituent cycles of MLbC are presented next.

4.1.1 ML organization cycle

The results highlighted the role of both management and technical teams in MLbC development and operation. Despite differences in sizes and responsibilities, the management team consistently played a pivotal role in coordinating, updating and monitoring the development process of MLbC.

The idea for this proof-of-concept came from the management team. After meeting with a vendor, they got really excited and pushed us to run this project. ∼Data Strategist

Simultaneously, the need for embedding technical skills within ML Organization and balancing them between both the management and technical teams was highlighted by our respondents as being of paramount importance.

Our hybrid model aims to achieve the best of a centralized and distributed organization. Our centrally coordinated Platform Team provides common capabilities to Service and Production Teams. Common capabilities refer to technical components and solutions that can be utilized by multiple teams in an organization or that require highly specialized expertise to produce. Good examples are the data models maintained by the Platform Team and the ready-made solutions on the infrastructure side. This approach supports both the autonomous work of the teams and a clear division of ownership and responsibility. ∼Manager (Data Platforms and Technologies)

Besides the inherent need to continuously reflect on the ML design, development and operation process, ML organizations were also dynamic and evolving. The ML Organization Cycle stood out as a characteristic that our respondents highlighted when explaining the differences between how ML Organizations are run in contrast to traditional IT.

Organizing AI efforts in a company is not a goal, but a constant search for balance. It is cyclic – a continuous exercise and development. ∼Head of AI

Initially, less mature ML Organizations had to handle an influx of new talent and rapid changes in the overall team size. These changes were clearly beyond the scope of MLOps frameworks and practices.

One of the challenges we face is figuring out how to ensure people follow the same principles, quality, and objectives when the team grows and changes multiple times in a short period of time. ∼Head of Analytics & Customer Insight

The methodology we follow gives us a framework, but not really details on how to run those steps as a team. ∼Data Strategist

Once more established, ML Organizations continued to evolve the repertoire of activities belonging to data scientists and other team members.

Ultimately, we want to get to an environment where most of the building blocks are maintained by the service providers. There is still too much DevOps work for us. We need to keep track of what junior data scientists are wrestling with and remove that. ∼Data Scientist

Even mature ML Organizations continued to change. High dynamism in working environments not only enabled but encouraged changes in the ways of working and organizing.

As we hoped, the development is much more agile with MLOps. However, we are not stopping here. We are constantly getting new ideas and making changes to improve ease and efficiency. ∼Data Scientist

We also found that there was no single optimal setup for an ML Organization in general. More surprisingly, the respondents acknowledged that, for a single firm, an ML Organization might circle back and forth between distributing and centralizing ML Organization.

After initially starting as a centralized data science team, we moved to a decentralized model. This did not last long, as we moved more toward a hybrid solution. Now, technology has advanced, and people have gained new skills, so we see a lot of new grassroot AI projects. To somehow control this growing chaos, it seems that we are moving back toward centralization. But that is perfectly normal. ∼Head of AI

4.1.2 ML technology cycle

The results indicated that artifacts, including data, ML models, infrastructure and software code, as well as their related configurations and processes, constituted ML Technology within an MLbC. We observed changes in these artifacts over time. Furthermore, the related configurations and processes exhibited dynamism throughout the development, release and operation phases. Hence, an ML Organization's primary responsibility was to manage, update and coordinate ML Technology Cycle, which corresponds with the primary focus of MLOps practitioners, as illustrated by the quote below.

From my side of things, Digital Engineering, what we end up working with a lot is some of the infrastructure pieces that are needed to support data engineering or data scientists' work. ∼Digital Engineering Lead

The respondents stressed that traditional IT resources and data alone were insufficient to build MLbC successfully.

The fact that you have a lot of data is not yet a sufficient starting point for the development of AI applications. I started the project with an attitude that we will conquer the world. In the end, I was much humbler. ∼Development Director

Furthermore, initiating the ML Technology Cycle required trial and error, where even established approaches to agile development faced challenges due to high levels of uncertainty.

When transitioning to MLOps we felt like riding a bicycle while building its wheels at the same time. Development sprints did not work because there was so much trial and error. We did not know exactly what it is that we were trying to build. ∼Data Scientist

Even after being established, the ML Technology Cycle was characterized by the need for flexibility and changes that allow ML Organizations to iterate ML model development and tackle various business problems.

All of the tools, except for BigQuery, are open source, so further development is possible and easy. Parts can also be replaced with commercial products if we find suitable ones in the future. ∼Data Scientist

Our setup enables the development of a wide variety of algorithms because the only requirement is that the model training and API jobs can be containerized. ∼Data Scientist.

The ability to cycle rapidly through technology development, implementation and operation characterizes a more mature ML Technology Cycle.

Agile environment and MLOps is about fast turnaround of ML ideas to products. We try as many new ideas as possible, because majority of iterations [sic] do not deliver good enough results. To find working ideas for actual ML use you have to be able to run lots of trials. ∼Data Scientist

Eventually, the high pace of ML Technology Cycle iterations formed one of the foundations for enhancing existing or creating new digital capabilities based on MLbC.

If we can speed up the cycling pace, then we have a chance to find what works. ∼Data Scientist

4.1.3 ML user cycle

The third building block of an MLbC was ML User Cycle. Our findings concur with MLOps view that ML Users directly interact with ML Organizations and play an essential role in initiating the demand for MLbC and evaluating their business impacts. However, the results of our study indicate that the ML Users Cycle is a much richer building block of the overall MLbC.

In our sample, we found that MLbC users were either external customers or users within the focal organization.

Our front page gets 500,000 unique visitors per day. So, our solution impacts a lot of consumers. ∼Head of AI

Some organizations could share many elements between several ML Technology Cycles to support MLbC that catered for both internal and external users.

We use the same chatbot platform to serve our internal users [employees] and customers. But, of course, the bots are trained on a different set of data and recognize different types of user intents. ∼Head of Robotics and Automation

This duality of ML User Cycle enabled ML Organizations to leverage the experiences from working on MLbC targeted to internal users when later creating new products or services for external customers.

We want to develop the service for internal users first. Let's test it out and improve the productivity of our own customer services. Once we figure out what works, we can productize it and sell [it] in a nice package to our business customers. ∼VP (Strategy & Development)

This form of cyclicality, however, was one of many we identified. Especially in cases where users were informed about ML-based artifacts being based on “artificial intelligence” or utilizing “cognitive technologies,” the users would probe and test the technology's capabilities.

It is really funny to see that the customers are testing the capabilities of our bot. They want to see where it works and where it fails. Later they might start using it like a search box and just input keywords. ∼Chatbot Product Owner

These interactions between ML Users and ML Technology shaped future use.

Some users were feeling ashamed that our bot could not understand what they meant. This negatively impacted the usage rate. ∼Senior Data Scientist

In some cases, the cyclicality of ML Users interacting with ML Technology led to learning and knowledge accumulation by the user.

[We] built a machine learning-based tool that enables [company name] [sic] airlines to predict possible disruptions to air traffic more accurately. Currently, they have a few guys who have been watching the weather forecasts and monitoring the air traffic for years, but they are going to retire soon. This tool captures their know-how and helps younger employees learn and do the job as if they had years of experience. ∼Head of Operations

4.2 Structural characteristics inhibiting alignment of machine learning-based solutions

The three constituent cycles of MLbC, which we identified in the previous section, interacted with each other throughout MLbC development, release and operation. The dynamic nature of these cycles, coupled with mutual dependencies and relationships, gave rise to structural characteristics of MLbC, which inhibited alignment.

The management may be willing and mentally ready to start dealing with AI, but [ …] no one in the management can name a single application. The operative employees understand what data they have and do not have, but they do not necessarily have the capability of assessing how it affects their competitiveness if they [make] bigger investments. This conflict is constant. [ …] We are talking about a technology that requires cooperation from many parties in the organization: the one who understands what is valuable in the business, then maybe someone who is more of a visionary—what is worth pursuing in a certain number of years–and then some technological people who can actually do that stuff, those who know the data, and so on. It requires cooperation from so many fronts. ∼Managing Director (AI consulting company)

We have categorized these Structural Characteristics under two sub-themes, Temporal Complexity and Context Sensitivity (Figure 7). Temporal Complexity stems from the high dependence of MLbC development, release and operation on the timing of events and activities involved in these efforts. Furthermore, the cyclical nature of the underlying building blocks of MLbC further complicates the temporal structure of the overall process. Context sensitivity relates to the vulnerability of MLbC development, release and operation to changes that might occur in the context of the focal task targeted by the MLbC, the focal organization or its external environment.

4.2.1 Temporal complexity

Apart from the temporal structure imposed by MLOps—the development, release and operation cycle—more nuanced topics related to time emerged. Our informants often referred to their experience of passage of time, to pace and to cyclicality as key to affecting MLbC's overall success.

If we can speed up the cycling pace, then we have a chance to find what works. ∼Data Scientist

A common cause of MLbC failure was unrealistic expectations of future users or management teams regarding how quickly or when the performance of ML technology will reach the targets.

The buzz around AI causes people to have overly optimistic expectations about results and how quickly they can be realized. ∼ VP (Product & Service Development)

We started working with [startup company name] two years ago. At that time, they had only one product, but this was not exactly what we wanted, so we asked them to build something new for us. Since then, I have been working crazy hours with their business development team in the US and Indian development team. Every time they trained a new model, they gave me a new set of results. Then, I checked all the errors and provided correct annotations on the contracts. After two years, we are now finally reaching the performance of 96% accuracy, which was our target before we can move to the rollout in the sales organization. ∼Senior Legal Counsel

However, our analysis revealed that data scientists or engineers, who represent ML Organizations, often focused too much on the cycling pace of the technology. By concentrating on technology, they overlooked the time required by others in the organization to carry out the non-technical tasks necessary for ML model re-training.

Technically, it does not take long to retrain the models. We thought that we would have a data flywheel, so that the performance keeps on improving as we accumulate more and more data. But in reality, we still need humans to annotate each datapoint. Time and availability of experts becomes the bottleneck. ∼Senior Data Scientist

Even within the ML Organization Cycle, temporal complexity was apparent. Fast-paced work and frequent changes were not welcome or easily aligned with the ways of working practiced by other people in technical roles within the organizations.

The pace of our work led to some backlash in the organization, especially SAS users. We call them the inquisition. They arranged for us a long workshop with focus on “governance” and “compliance”. ∼Data Scientist

Those team members who did not work full-time within MLOps had challenges to keep up. The direction of work has been changing so quickly that when they came back to the office the following week others were already working on a new idea. ∼Data Scientist

Temporal complexity was also related to user interactions. Diversity of users frequently called for increased diversity in the ML Organization, resulting in delays caused by communication challenges.

Even the implementation teams are increasingly multidisciplinary. But looking at things from a variety of perspectives easily brings with it communication challenges. It takes time for people with different backgrounds and objectives to understand each other. ∼AI Consultant

Furthermore, the personalization of MLbC behavior created challenges because of differences in the temporal preferences of different users.

The level of personalization is really complicated to implement. It is not only about the content that needs to be personalized, but also about the timing of information delivery. Some people want to get notifications immediately, regardless of the time of the day. Others get angry when they get messages outside of the office hours. Some do not want to have anything extra appearing on their mobile ever. ∼Head of Digitalization

Finally, the nature of ML applications in use frequently involves an element of forecasting. Making decisions or carrying out actions based on ML-generated forecasts, projections or recommendations often add another layer to temporal complexity.

When we make the decisions about routing, where the deliveries should be unloaded, at that time we still do not know how much available capacity there will be at the target destinations. This is the type of forecasting problem where we can make a big impact on customer experience. ∼Data and Automation Lead

4.2.2 Context sensitivity

Another form of structural characteristic of MLbC deflecting alignment efforts was Context Sensitivity, which is rooted in the fact that ML models are not explicitly programmed. Rather, they are trained based on data that may or may not capture the true underlying reality across all contexts in which MLbC will operate. Our findings reveal three dimensions of context affecting MLbC development and alignment: task context, organizational context and external environment.

Our respondents recognized that MLbC constituted a context-sensitive structure within which they were forced to work. As a result, they used context to define and scope the development efforts.

Context is everything. What we do internally [to deal with it] is we split the data into smaller pieces, because it is much easier to align and train an ML model when you have smaller context. ∼CEO (AI Start-up)

In some cases, if the sensitivity of an MLbC to task context needs to be understood early enough and across the ML User and ML Organization cycles, then the development might be abandoned altogether, and the investment is completely lost. This type of sensitivity is a hallmark of misalignment between the statistical distribution of the training data and the distribution of the data that feeds into models during operation.

The solution relying on ML, which we built, performed really well most of the time. However, in the last meeting before the planned implementation our big boss said that we cannot accept out of whack results generated in some contexts. The whole project was frozen after that. ∼Software Developer

This type of sensitivity prevails even for MLbC, which has successfully operated for some time. Our respondents identified that Task Context Changes could disturb the alignment with operations. The resulting misalignment often revolved around data deficiencies.

Every time we get a new customer onboard the [ML-based solution] performance drops for some time – until we get new training data from that customer. ∼Service Manager

ML Organizations might also encounter this type of sensitivity to task context differences earlier in the development process, which manifests through difficulties in obtaining sufficient training data for some task contexts, as illustrated by the following quote.

It was a surprise to us that the development process for the same solution would differ so much depending on whether the users were our own employees or consumers. Getting consumers to try new stuff that we build is easy. But internally our people are so busy that it is hard to get them to actively pilot the new app and give us feedback. ∼Head of Digitalization

Another form of context sensitivity is related to Organizational Context Change. Changes occurring in other parts of the organization could directly impact the data flowing into ML models.

[ …] this morning a colleague told me that the latest dashboards look very, very weird. [ …] he went to R&D and found out that they changed the logic of how data is transmitted from the machines. […] it has totally, totally blown out the logic on the other side. But that is daily life. ∼Head of Data Science

However, organizational changes could also impact MLbC by changing not the technology but rather the people behind the technology—that is the ML Organization Cycle.

The UI [user interface] personalization that we have developed boosted user engagement KPIs [Key Performance Indicators]. Everyone was pumped about this. But once guys from [vendor name] were no longer around and we had a re-org, there was nobody to keep the project alive. The UI designers went back to the manual approach. ∼Head of AI

Finally, sensitivity to Changes in External Environment Context also could misalign an existing MLbC. Major shocks, such as the COVID-19 pandemic, can undermine MLbC's usefulness.

Obviously, now [after the breakout of the COVID-19 pandemic] all of the predictive models are broken because the external context has changed. ∼Data Strategist

However, not only can major shocks affect MLbC-business alignment, but in some cases, gradual and subtle changes in the external environment erode and undermine existing MLbC performance.

Once the [NLP-based contract analysis] tool achieves good enough performance level, [ …] periodic retraining might be needed to adapt to new regulations, such as introduction of GDPR recently, or changes in contract drafting styles or standards. ∼Corporate Legal Counsel

4.3 Practices for machine learning-based capability alignment

Our findings suggested that the structural characteristics of MLbC, covered in the previous section, inhibited alignment of the solutions throughout the development, release and operation phases within the organizations included in our study. Despite the alignment challenges rendered by these characteristics, some organizations succeeded in their MLbC efforts and, thus, enhanced existing or developed new digital capabilities. Our analysis also revealed that such organizations leveraged two alignment practices: Fostering Temporal Congruence and Cultivating Organizational Meta-learning (Figure 8). We found that the former set of practices primarily addressed Temporal Complexity, while the latter Context Sensitivity.

4.3.1 Fostering temporal congruence

Since our respondents recognized the importance of Temporal Complexity in shaping and constraining MLbC development, release and operation efforts, they frequently discussed their approaches to addressing the resulting challenges. First, they engaged in a synchronization of development efforts that involved filtering incoming requests from the focal organization and users to the ML Organization.

Part of my role is also to filter the requests that are coming to data scientists. Without such buffering they would spend too much time on dead-end projects. ∼ Director (Data-Driven Services)

This allowed more optimal time utilization by data scientists, which most ML Organizations considered a scarce resource. Furthermore, because of this optimization, ML Organizations could more rapidly respond to high-priority requests, thus increasing the overall alignment of ML Organization and ML User Cycles.

Another form of synchronization was required to align the pace and progress between ML Organization and other parts of the organization. Successful synchronization frequently prevented MLbC implementation failure or maintained high morale levels in the ML Organization.

Our team has built a mobile app with state-of-the-art speech recognition capabilities for [large corporation name] in a few weeks. After that it would take them ages to take it in front of their customers. We would have given up on this project if it were not for their business development manager. He kept the project alive for 9 months of internal legal approvals. Now, they have launched the app already and we keep on iterating and making it better for consumers. ∼CEO (AI start-up)

Data scientists like to see their models go into production and create impact. With slow IT organization it might take some time though. There is some buffering needed. If not, especially, the younger team members might leave after some time. Therefore, we decided to give more ownership to data scientists. This way they do not need to send tickets all the time to get something fixed. ∼Senior Data Scientist

We observed that the need for additional synchronization decreased once alignment between the constituent cycles of an MLbC was achieved and sustained after entry into operation. Successful organizations benefited from the faster iteration pace of the overall MLbC development and operation, as well as the apprehension and appreciation of ML applications by ML Users.

We are in sync and cooperate better with other business units because the development is faster, and we can offer more insight into how the models work. ∼Data Scientist

Forging New Roles emerged as another practice fostering temporal congruence. These new roles were instrumental in removing or alleviating delays and time lags that hampered the progress of or slowed down the cycling pace within either individual steps or entire phases of development, release or operation. These new roles frequently resided within the ML Organization Cycle, such as MLbC Product Owner or ML Trainer. People taking on these roles often had previous experience either as a user or working with external users. These insights and understanding were essential foundations of this MLbC alignment practice.

We have realized that re-training our bots is not something that we can do every now and then. It is an ongoing effort. So, we have assigned some of our agents to become full-time bot trainers. ∼Head of Robotics and Automation

Being an ML product owner is a full-time job. You need to dedicate enough time to it. ∼Data Scientist

However, in some cases, the alignment resulting from forging new roles progressed even further. To address temporal complexity, some organizations in our study explicitly increased integration between ML Organization and ML User cycles. Some of our respondents explained that their ML Organizations actively recruited and trained “ambassadors” within the ML User Cycle. These “ambassadors” continued in their previous roles but also assumed new responsibilities related to providing timely, targeted and frequent feedback on issues pertaining to MLbC. In this way, they could alleviate some challenges related to the mismatch in timing and pace between the constituent cycles.

The whole idea behind the “AI Ambassador Program” is about speed and timing. The success of the ambassadors relies on them serving as a bridge between business units and technology. We do not have enough time and resources within the Data Science Team to constantly educate and encourage people to work with data or ML-based services. Business leaders and users need frequent and instant feedback on AI issues. That is what ambassadors can provide. ∼Business Development Manager for Robotics & AI

The third type of practice fostering temporal congruence revolved around Expectations Management. Apart from managing ML Users' expectations regarding ML Technology's performance, our informants recognized the importance of managing expectations concerning the time required to reach that performance.

We try to avoid these situations where the expectations and the actual outputs are not aligned. This is typically the case when they [internal customers from business units] have high-flying ideas that would take a lot of time to build. ∼Data Strategist

However, the expectations were not unidirectional. Many of the respondents representing ML Organizations strongly also argued in favor of setting clear expectations for ML Users. They emphasized that the speed of changes within the ML User Cycle also drives the time required to develop, release and operate the MLbC. Thus, managing expectations involved clarifying and validating the ability of the ML Users to change their own cycling pace.

One of the very important points that we validate is ML solution utilization. Far too often people are just interested in some results. But this is not enough for us. We need to understand the current business process and how the solution is changing the process. This enables us to create business benefits. ∼Data Strategist

It is easy for people in the organization to propose ideas about what could be useful. But we need to bring to the discussion which roles, task[s], and maybe responsibilities will change. We need to have a common language and understanding of the changes and impact. ∼Head of AI Center of Excellence

Congruence between the expectations on both sides rendered the environment in which rapid experimentation was possible. This increased the pace of moving from MLbC development to continuous adjustment to ensure MLbC-business alignment.

To meet the expectation of creating impact, you actually need to fail a lot and fast. What I mean by that is we are not committing to one idea, but rather trying out quickly a portfolio of approaches. Most will fail and everyone needs to expect that. But some will help us optimize the UI behavior based on user profiles. ∼Data Scientist

4.3.2 Cultivating organizational meta-learning

Our respondents identified two practices, Learning to Learn and Embracing Data Work, which enabled successful MLbC development, release and operation despite Context Sensitivity. We have grouped these two practices under the umbrella term of Cultivating Organizational Meta-Learning. In the context of this study, meta-learning refers to the process by which organizations increasingly understood and orchestrated their own growth and learning with respect to MLbC [1], which corresponds to the definition proposed by Lei et al. (1996, p. 562):

Meta-learning is the simultaneous conceptualization of different and contradictory forms of knowledge. It integrates information transfers, experimentation, and dynamic routines into a systemic perspective. Meta-learning may create additions to, or substitutions of knowledge (new replacing outmoded knowledge).

The practice of Learning to Learn relied on the gradual accumulation of understanding and experience within each of MLbC's three constituent cycles and, more importantly, across them. By experiencing context changes and devising ways to respond to them, successful organizations progressed beyond the initial MLbC development and moved toward alignment.

The potential for AI applications is identified everywhere and continuously. But you cannot push the technology. The implementation needs to happen in light of common learnings. ∼VP (Product and Service Development)

One of the distinguishing features of Learning to Learn practices was the employment of multiple and coordinated efforts to raise overall organizational knowledge about MLbC. This often involved starting broader educational campaigns for people across organizational levels on ML-related topics.

It was a deliberate strategy to run multiple education efforts across levels in the company. All our top-level managers and 30 mid-level managers went through an executive education program focused on AI. 150 regular employees were trained in the basics of AI. We also run many focused workshops, internal hackathons, and actively benchmark against other companies and participate in the ecosystem activities. ∼Director (Data and Automation)

Such campaigns not only enabled future ML Users to communicate more clearly regarding the task and environmental context with ML Organization but also reduced the sensitivity of MLbC to organizational context changes. Our respondents were clear that to increase alignment with MLbC, the educational campaigns needed to be combined with more targeted learning at the intersection of the ML User and ML Organization Cycles. For example, data science teams educated other people involved in technical work, thus increasing their ability to learn and accelerating the learning rate.

It is a continuous process to get other parts of the organization aligned and capable of understanding how to use ML. For example, we are continuously onboarding several people at a time from Business Intelligence. ∼Data Scientist

At the same time, data scientists and other members of an ML Organization had to continuously develop new ways to acquire knowledge about MLbC's non-technical aspects. This meant improving their grasp of either industrial- and business-use cases or consumer behavior, thus contributing to task and environmental context understanding.

For us it is important that data scientists gain as much domain understanding as quickly as possible. ∼Data Strategist

Make it a habit to spend a day now and then in the front lines. Experiencing problems firsthand is often the best way for getting a non-biased view on any issue. ∼Director (Data and Automation)

The resulting cross-sensitization between the ML Organization and ML Users increased MLbC's robustness to context change because of the higher awareness of existing and potential context sensitivities. Learning to Learn also leveraged an exchange of insights between people within their respective cycles.

Let’s bring together people interested in ML from around the organization. Let’s put them on a learning and growth path. Then, we give them the credibility and mandate to share the learnings in their own teams. ∼Data Scientist

Finally, the inherent experimental nature of an MLbC cycle, as discussed when presenting ML Technology Cycle, lends itself to supporting a Learning to Learn practice. This, however, cannot be done in isolation from the other cycles if it is to reduce sensitivity to context changes.

Embracing Data Work was the other form of organizational meta-learning practice discussed by our respondents. We can conceptualize data work as agile, multilayer, cross-functional efforts to acquire, process and utilize data in support of meta-learning. The centrality of data to MLbC's technical and business viability and their alignment was a recurring theme. Our respondents recognized that working with data was their primary tool when handling the sensitivity of MLbC to changes in task, environmental and organizational contexts.

Ambiguity around the ML-based solutions is always about the data. We cannot know for sure if it is feasible before we know the data. ∼Data Scientist

Hence, most mature organizations in this respect continuously developed new and expanded existing ways of working with the data and data sources. Increased volume and pace of data flow enabled them to leverage existing data resources in the development and operation of MLbC. This, in turn, alleviated some of the context sensitivities.

The big departure from old ways of working and transition to MLOps paid off. It is reflected also in the approach that other teams take when working with data. We see increased interest in cloud usage and a surge in internal requests for APIs. ∼Data Scientist

However, technical data work elements are needed to fully represent this MLbC alignment practice. Some respondents indicated that it is not just about improving data flow in general. Rather, the data needs to be of as good a quality as possible and represent input for ML model training, which will deliver the most improvement, given the operational context.

I am a quality-oriented person, and I can use that skill to focus on where we can help our users improve the ways of working with [ML-based solution name]. Guiding the users is part of the process of model training. ∼Data Operator

In ML huge volume of background data is important, but the most value is in your own domain data. In practice, we only add our own data whenever we do regular model retraining. ∼CEO (AI start-up)

These views were also counterbalanced by respondents who emphasized that alignment of MLbC through working with data is resource intensive. Thus, the context sensitivity of MLbC is often inversely proportional to the resource commitment of the focal organization to Embracing Data Work.

I try to avoid jumping too quickly into the data, because that is where the time and resource usage starts getting heavy. ∼Data Scientist

5. Discussion

Figure 9 synthesizes our findings on how new or enhanced MLbC were developed by the organizations we studied. We provide a glossary of key terms in Table 3. This model portrays two essential mechanisms enabling MLbC development and alignment practices. First, organizations advance through this process by progressively and iteratively (Re)Initiating, Effectuating and Sustaining Alignment between their ML Organization, ML Technology and ML User Cycles. Second, the underlying mechanism of this process is governed by the interaction of two pairs of Structural Characteristics of MLbC and Alignment Practices: Temporal Complexity and Fostering Temporal Congruence and Context Sensitivity and Cultivating Organizational Meta-learning, respectively.

Organizations must (Re)Initiate Alignment when developing MLbC because, at this stage, novel elements within the constituent cycles, or their new versions, are being put together. This triggers the need for Fostering Temporal Congruence to tackle Temporal Complexity. This mechanism is particularly salient in the development phase. Without synchronizing efforts and managing expectations, MLbC development is likely to fail due to differences in the cycling pace of ML Technology development and those of ML Organization and ML User. In many cases, (re)initiating alignment also requires forging new roles in organizations. Furthermore, at this initial phase, organizations are also learning about potential or newly discovered sensitivities of the envisioned MLbC to changes in context. This implies that members of an ML Organization and ML User cycles need to continuously evolve their own knowledge and understanding to enable learning of algorithms. Thus, intensive preparatory data work is a common denominator across MLbC development work at this stage.

Effectuating Alignment phase coincides with release, when the alignment between the constituent cycles undergoes a real-world application. It is also the culmination time for practices fostering temporal congruence. Only through a synchronization of efforts can this phase eventually succeed. Also, this is when the expectations of ML users and members of ML Organization are cleared against each other and the actual performance and outcomes. Meanwhile, some new roles created at the earlier phase will need to achieve maturity or, at least, a degree of routinization. Effectuating alignment is akin to learning because it involves moving ML-based technology from development to production. Equally, this means starting to work with “live” data and impacting users.

Nevertheless, sustaining alignment is required in response to constant changes and cycling within at least one, if not all, of the constituent cycles. During the sustaining alignment phase, temporal complexity should already be controlled by deploying practices that foster temporal congruence. Yet, MLbC's context sensitivity retains its importance. Successful sustaining of alignment of MLbC is dependent on remaining vigilant about changes that might occur in the context of the focal task, the organizational process and external environment. Thus, MLbC at the operational phase requires organizations to cultivate meta-learning. The need for additional learning might translate into either continuous data work or cycling back to re-initiation or effectuation of alignment phases.

5.1 Theoretical implications

Insights from our work advance the emerging IS literature on organizational capabilities enabled by ML technologies (Benbya et al., 2021; Berente et al., 2021; Mikalef and Gupta, 2021; Zhang et al., 2021b). By disentangling the process of MLbC development, the inherently complex relationships of ML Organization, ML Technology and ML User cycles over time, and related ML-based digitalization of organizations we respond to Raisch and Krakowski's (2021) call for complexifying theorization of ML-related phenomena to grasp their richness. Furthermore, our theorization directly addresses the question of “how” MLbC are developed, identified as a notable gap in our understanding of MLbC (Mikalef and Gupta, 2021). Overall, our insights support the view that the development of MLbC further contributes to already complex phenomenon of organizational digitalization (Mucha and Seppälä, 2021).

More specifically, the first contribution of our work stems from bridging the gap between research on MLOps and IS literature. The emergence and rapid progress of MLOps have been increasingly noted by researchers and practitioners (Aguilar Melgar et al., 2021; Baier et al., 2019; Choudhary et al., 2022; Kolltveit and Li, 2022; Lwakatare et al., 2020a, b; Muralidhar et al., 2021; Paterson et al., 2021; Renggli et al., 2021). However, these efforts mainly led to a better understanding of the technical challenges pertaining to ML-based artifacts. While this growing literature indicates organizational challenges (Mäkinen et al., 2021), the evident gap between ML-based applications—the outcome of MLOps—and MLbC has been overlooked. This is somewhat surprising, given that IS scholars frequently note the need to align technology, processes and people. While our results align with prior IS studies, such as Lyytinen et al. (2020), we bring attention to the overlooked temporal and cyclical aspects of that alignment. We highlight that alignment evolves and iterates through (re)initiating, effectuating and sustaining phases and is governed by the interaction of MLbC's structural characteristics and respective alignment practices.

Contributing to AI management literature, our second contribution adds nuance to the temporality of MLbC development. Past research has already identified that the development of MLbC depends on an organization's ability to select, orchestrate and leverage specific tangible, human and intangible resources (Mikalef and Gupta, 2021; Zhang et al., 2021b). Yet, knowing which resources are needed is only a first step in advancing our understanding of MLbC development. Our study shed light on how these resources could be combined and governed, as well as how this process leads to the creation of MLbC over time. Therefore, we facilitate a meaningful IS conversation on MLbC by bringing contextually rich and fine-grained insight and providing a much-needed explanation of mechanisms governing that process. Moreover, our findings indicate that prior conceptualization of MLbC (e.g. Murray et al., 2021) missed a critical piece of the alignment puzzle. Namely, alignment is not driven solely by the type of ML Technology underlying MLbC; instead, MLbC-business alignment is (re)initiated, effectuated and sustained through practices of synchronizing efforts, forging new roles, managing expectations, learning to learn and embracing data work and their joint interaction with structural characteristics of MLbC. Thus, it “takes richness to grasp richness,” as Weick (2007, p. 16) famously stated.

Finally, we contribute to RBV and IT-business alignment literature, which equipped us with the theoretical lens for this study. The process of resource structuring, bundling resources into capabilities and eventually leveraging these capabilities by organizations has primarily been researched on the organizational or top management level (Sirmon et al., 2011). However, organizations presently use ML technologies that are predominantly narrow in scope with limited capabilities (Benbya et al., 2020). Hence, our research on MLbC development brings insights into the understudied topic of capability development within organizations. Additionally, IT-business alignment literature has neglected the temporal and process perspective on alignment (Chan and Reich, 2007), despite the widely acknowledged processual nature of IT-business alignment (Luftman et al., 2017). More broadly, our study suggests that IT-business alignment should be revisited as the definition of IT has evolved in the age of AI. Furthermore, RBV and IT-business alignment emerged during an era of deterministic technologies, where humans had to explicitly codify instructions for how technology should work. Now, however, with the advent of modern ML technologies, organizations are no longer bound to that approach and must deal with stochastic technologies that are learning and changing, even without human interventions (Ågerfalk, 2020; Berente et al., 2021; Lyytinen et al., 2020). This challenges some long-held beliefs, especially in IT-business alignment research, that assumes technology is relatively fixed (Chan and Reich, 2007). Therefore, our study contributes insights into this novel area at the intersection of organizational capabilities, IT-business alignment and non-deterministic technologies.

5.2 Practical implications

Insights from our research have direct practical implications. First, managers should recognize that creating a new MLbC is not a one-off assignment but a continuous and multifaceted process (Mucha and Seppälä, 2021). This suggests that considering the timing of decisions, actions and their outcomes, as well as the pace at which iterations happen, is crucial for success. Second, recognizing context sensitivity is vital given the volatile environment in which today's organizations operate. MLbC are inherently sensitive to context changes; thus, in response, managers need to cultivate meta-learning in their organizations. This requires data work in the early phases of ML-algorithm training and continuing these efforts to refine and discover new data needs. They must also create an environment where employees are encouraged to learn because learning cannot be left to machines alone. Third, managers should consider and welcome cyclicality within all three MLbC constituent cycles. While the need for iterations in technology development is widely acknowledged, far less attention is dedicated to the continuous evolution of ML Organization and ML Users. Overlooking constant changes within these two latter cycles undermines alignment, thus the overall viability and sustainability of MLbC. Furthermore, individual MLbC alignment practices identified in our work and the empirical examples provided can inspire organizational leaders on how to deal with MLbC's structural characteristics.

6. Limitations and future research directions

This study is among early empirical attempts to understand and explain MLbC development in organizations beyond MLOps's prescriptions. We applied a qualitative case study research methodology to an extensive data set to gain insight into the emergence and routinization of the focal process within many organizations. We developed an empirical model and exhibited the transferability of findings across organizations. This study, nevertheless, faces some limitations which invite further research. First, most organizations participating in the study were well-established corporations or successful startups with large customer bases. This indicates a need to gain insight into MLbC development for companies at their establishment and early stages. This topic might be of increasing importance, as many start-up companies develop MLbC as the core of their offering and, potentially, competitive advantage. Second, despite our efforts to triangulate the results by incorporating data from countries outside Finland, most of the materials used in the analysis originated from that country. This means that validating our model in other countries with different organizational cultures is warranted. Third, our work identified structural characteristics of MLbC that are vital to consider when developing organizational capabilities and related alignment practices. However, detailed understanding of the relative importance of these across temporal and contextual dimensions is left for future research. Fourth, the fields of ML and, therefore, MLOps are rapidly evolving. This means that radical advances in ML technology and related organizational practices might bring novel challenges and opportunities for organizations developing MLbC. Consequently, future replication studies and further extensions of our work might be needed. Fifth, a detailed analysis of the technical aspects of MLbC was not within the scope of our work. Further research is thus necessary to account for different types of ML technology and their application and appropriation. Lastly, due to the nature of our data, our analysis was limited to organizational boundaries. However, MLbC development is not only affected by organizational factors. Therefore, future work can investigate how social, economic, legal or political factors may affect MLbC development and alignment.

7. Conclusion

This study extends research on digital transformation and digitalization of organizations by revisiting the notion of MLbC from a new perspective. We argue that the development of MLbC is more complex than MLOps or similar frameworks could fully capture. Grounded in theory and an extensive empirical study, we propose an overarching framework representing three main components of MLbC development—ML Organization, ML Technology and ML User—and three phases of MLbC development—Development, Release and Operation. We then theorize how structural characteristics of MLbC—Temporal Complexity and Context Sensitivity—challenge the alignment process—(Re)Initiating Alignment, Effectuating Alignment, and Sustaining Alignment—and accordingly introduce Fostering Temporal Congruence and Cultivating Organizational Meta-learning as two MLbC alignment practices. Metaphorically, we see these two alignment practices as bicycle wheels that need to be built by organizations attempting to cycle through (re)initiating, effectuating and sustaining the alignment.

Figures

Figure 1

Rapid growth trends in the popularity of MLOps in research and practice

Figure 2

Research procedure

Figure 3

Data structure (Part 1)

Figure 4

Data structure (Part 2)

Figure 5

Data structure (Part 3)

Figure 6

Three constituent cycles of MLbC and their gradual alignment through the development, release and operation phases of MLOps

Figure 7

Structural characteristics of MLbC

Figure 8

MLbC alignment practices

Figure 9

Machine learning-based capability development and alignment process model

Table 1

Literature review summary table

Key topics	Research on MLOps	Research on organizational use of ML
Capability Development and Time Capability development is an iterative process unfolding over time (Ancona et al., 2001; Chan and Reich, 2007; Helfat and Peteraf, 2003; Luftman et al., 2017)	• MLbC Development Iteration: MLOps is viewed as a process with multiple iterating components (Amershi et al., 2019; Mäkinen et al., 2021; Renggli et al., 2021; Zhou et al., 2020) • MLbC Development Cyclicality: MLOps is cyclical and with multiple feedback loops forming over time (Lwakatare et al., 2020a; Mäkinen et al., 2021; Mucha et al., 2022; Paterson et al., 2021; Renggli et al., 2021) • MLbC Development Timing, Pace and Synchroneity: Focus on the temporal aspects centers on technology development (e.g. updating models in time), and operation (e.g. collecting feedback in time) at application-level (Huang et al., 2022) and overlooks the temporality, such as timing, pace and synchronization within the broader organization and user context (Muralidhar et al., 2021)	• Organizational resources needed to develop MLbC have been identified in a static manner, thus leaving process and temporal aspects for future investigation (Mikalef and Gupta, 2021) • The importance of recognizing temporal progression and feedback loops in the process is documented well (Benbya et al., 2021; Mucha et al., 2023a, b; Raisch and Krakowski, 2021; Sturm et al., 2021a) • Multiple iterations and feedback loops, as well as complexities and uncertainties related to timing, pace and rhythm of developments, have been noted in recent qualitative studies (van den Broek et al., 2019; Grønsund and Aanestad, 2020; Ruissalo et al., 2022; Sturm et al., 2021b; Waardenburg et al., 2022). However, these studies primarily concentrate on the development and release phases, lacking comprehensive insights into the operational phase
Capability Development and Organizational Alignment Capability development requires alignment of multiple actors and organizational (sub)structures (Barney and Felin, 2013; Bharadwaj, 2000; Chan and Reich, 2007; Gavetti, 2005; Goodhue and Thompson, 1995; Grant, 2016; Henderson and Venkatraman, 1999; Luftman et al., 2017; Lyytinen and Newman, 2008; Sirmon et al., 2011; Trang et al., 2022)	• MLbC Development and Integration: MLOps at scale requires deep integration between ML development process and organizational processes (Mäkinen et al., 2021). Current MLOps practices typically fail to align and integrate with broader organizations due to, for example, the absence of a systematic ML pipeline (Moreschini et al., 2022) with context-specific solutions (Garcia et al., 2018), the lack of knowledge on governing ML-based solutions (Lwakatare et al., 2020a) and delegation mechanisms (Baird and Maruping, 2021) • MLbC Development and Sociotechnical Challenges: Recognized sociotechnical challenges related to the MLOps-business alignment include, among others, establishing users' trust, increasing explainability, harnessing controllability and facilitating self-adaptation (Abedin, 2022; Laato et al., 2022; Martínez-Fernández et al., 2021)	• The human roles and structures within organizations are important drivers of organizational ML development and use (Mikalef and Gupta, 2021; Mucha et al., 2023b; Strich et al., 2021) • New organizational roles, both managerial- and operational-level, emerge following the implementation of MLbC (Berente et al., 2021; Grønsund and Aanestad, 2020; Strich et al., 2021; Waardenburg et al., 2022) • The aspect of changes within the team or organizational sub-unit developing and operating ML technology and their influence on MLbC has received only very limited attention (Fountaine et al., 2019)
Capability Development and Context Capability development depends on both internal and external setting and is sensitive to context changes (Chan and Reich, 2007; Gavetti, 2005; Lyytinen and Newman, 2008; Sirmon et al., 2007; Wilden et al., 2016)	• Contextual Oversimplification of MLbC Development: MLOps frameworks assume that the process of solution adaptation to operational context changes can be simplified through data processing and automated model re-training (Choudhary et al., 2022; Paterson et al., 2021) • Contextual Monitoring of MLbC Development: Post-implementation, ML models need to be carefully monitored by human actors (Lyytinen et al., 2020; Symeonidis et al., 2022) due to model's error-prone nature (Asatiani et al., 2021) • Contextual Coordination of MLbC Development: The monitoring and intervention processes might require a significant amount of coordination and operational employees' involvement to capture the reality of new context beyond the scope of current MLOps practices (Pääkkönen et al., 2020)	• Empirical studies recognize the sensitivity of MLbC development to external context, such as fluctuations in consumer buying patterns (Zhang et al., 2021b), prices of raw materials (Min et al., 2019), customer profiles (Mucha et al., 2023a) and stock market trends (Sturm et al., 2021b) • Conceptual work recognizes the importance of context changes, but there is no agreement regarding the consequences of these changes - Augmenting human decision-making with ML is likely to be more beneficial when the external context changes at a high rate or magnitude (Balasubramanian et al., 2022) - A simulation study indicates that in volatile environments the role of people is essential in searching for new knowledge and actively (re)training ML models (Sturm et al., 2021a)

Source(s): Author's own creation/work

Table 2

Data collection

Phase	Source	Method	Topics	Purpose
Primary study data August 2018 – December 2021	Finland's AI Accelerator (FAIA)	Participatory observation (N = 123) and interviews (N = 6) during FAIA team's internal meetings, workshops with the participating companies, and events arranged by FAIA (about 162 h of events in total, of which about 61 h audio recorded and transcribed) FAIA's internal documentation (N = 62)	• Actors, technologies, tasks, and organizational structures involved in organizational ML initiatives • Organizational practices, approaches, and responses related to ML initiatives • Challenges and changes over time in relation to the topics listed above • Outcomes of ML initiatives	• Identifying the key themes for the organizational development of MLbC • Cross-comparison of different cases of MLbC development and deployment • Tracking that process over time and across multiple organizations to enable cross-comparison leading to comprehensive understanding
Confirmatory data April 2019 – January 2022	Non-FAIA affiliated organizations	Semi-structured interviews with employees of non-FAIA affiliated organizations (N = 20, about 13.5 h of audio recording in total, transcribed)	• Actors, technologies, tasks, and organizational structures involved in organizational ML initiatives • Organizational practices, approaches, and responses related to ML initiatives	• Verifying the key themes for the organizational development and deployment of MLbC • Enriching data collection to allow triangulation based on data from outside of the primary study setting
Contextual immersion August 2018–June 2023	Practitioner sources	Topic mining from practitioner sources (Gartner, MIT Technology Review, HBR, etc.) and ML-related podcasts (AI in Business, TWIML, Eye On AI, Practical AI, etc.)	• AI/ML-related activities, processes, challenges, and outcomes in organizations across industries and geographies • MLOps frameworks and practical experiences related to MLOps	• Sensitizing researchers to the emerging concepts and latest technical developments in organizational practices related to MLbC development

Source(s): Author's own creation/work

Table 3

Definitions of key terms

Building block		Definition
ML-based Capabilities (MLbC)		An organizational ability to align ML-specific and other resources to perform a particular organizational activity (set of tasks) in a reliable, repeatable and value-added manner
The Constituents of MLbC	ML Organization Cycle	A series of successive states, structures and functional roles pertaining to a temporary or permanent social subsystem formed under the umbrella of the focal organization. ML Organization controls resources, enacts processes, sets rules, plans objectives and possesses key competencies related to MLbC development, release and operation
	ML Technology Cycle	A series of successive states, configurations and processes pertaining to artifacts, including data, ML models, infrastructure and software code related to MLbC development, release and operation
	ML User Cycle	A series of successive states and functional roles of organizational or external human agents who directly interact with or benefit from MLbC
MLbC Alignment Phases	(Re)Initiating Alignment	A phase in MLbC development process when novel elements within the ML Organization, User and Technology cycles (their new versions), are designed for and directed toward alignment with business goals and strategies. This phase typically corresponds with the Development phase of MLOps
	Effectuating Alignment	A phase in MLbC development process when ML Organization, User, and Technology cycles meet each other to support the successful transition of ML-based application from development environment to operation. This phase typically corresponds with the Release phase of MLOps
	Sustaining Alignment	A phase in MLbC development process when the constituent cycles of MLbC are kept in alignment through agile, multilayer, cross-functional efforts to acquire, process, and utilize data, and reflect on the accumulated learnings in relation to context changes. This phase typically corresponds with the Operation phase of MLOps
Structural Characteristics of MLbC	Temporal Complexity	The presence of high dependence between timing of decisions, actions, and outcomes within the development, release, and operation phases of MLbC, pace at which constituent cycles iterate and their effects on the MLbC development process and its outcomes
	Context Sensitivity	The high vulnerability of MLbC to changes that might take place in the context of focal task (e.g. specific business process), focal organization (e.g. changes in the team developing ML model) or its external environment (e.g. market)
MLbC Alignment Practices	Fostering Temporal Congruence	Organizational practices primarily addressing Temporal Complexity by improving the alignment of the constituent cycles of MLbC through synchronizing efforts, forging new organizational roles and managing expectations
	Cultivating Organizational Meta-learning	Organizational practices primarily addressing Context Sensitivity by improving the alignment of the constituent cycles of MLbC development through organizational learning to learn and embracing data work

Source(s): Author's own creation/work

Table A1

Summary of participants' profiles

Group	Individual person id	Number of events where the individual person participated (out of 149 events)	Exemplar titles/roles (in some cases more than one individual person used the same title)
PHASE: Primary study setting SOURCE: Finland's AI Accelerator (FAIA)	P1	117	Accelerator Lead, AI Consultant, AI Consultant (Data Privacy), AI Consultant (Development and Implementation), AI Consultant (Head of Operations), AI Consultant (Lead AI Scientist), AI Consultant (Organization and Culture), Business Development Manager for Robotics & AI, CEO (AI consulting firm), CEO (AI start-up), Chatbot Product Owner, Communication Specialist, Corporate Legal Counsel, Customer Service Manager, Data and Automation Lead, Data Operator, Data Scientist, Data Strategy Consultant, Development Manager (Digitalization), Director (Data and Automation), Director (Data-Driven Services), Head of AI, Head of AI Center of Excellence, Head Of Analytics and Customer Insight, Head of Data Science, Head of IT Interfaces, Head of Operations, Head of Partnerships, Head of R&D, Head of Robotics and Automation, Manager (Data Platforms and Technologies), Managing Director (AI consulting company), Market Researcher, Professor of Computer Science, Senior Data Scientist, Senior Legal Counsel, Service Manager, Software Developer, VP (Product and Service Development), VP (Strategy and Development)
	P2	98
	P3	9
	P4	29
	P5	13
	P6	29
	P7	2
	P8	24
	P9	27
	P10	1
	P11	2
	P12	3
	P13	2
	P14	2
	P15	2
	P16	1
	P17	2
	P18	3
	P19	1
	P20	1
	P21	1
	P22	3
	P23	6
	P24	4
	P25	4
	P26	4
	P27	3
	P28	7
	P29	7
	P30	4
	P31	7
	P32	5
	P33	5
	P34	6
	P35	5
	P36	7
	P37	3
	P38	6
	P39	2
	P40	1
	P41	1
	P42	6
	P43	6
	P44	4
	P45	5
	P46	5
	P47	3
	P48	5
	P49	5
	P50	5
	P51	6
	P52	2
	P53	2
	P54	2
	P55	2
	P56	2
	P57	2
	P58	2
	P59	2
	P60	1
	P61	2
	P62	2
	P63	6
	P64	7
	P65	7
	P66	6
	P67	6
	P68	7
	P69	5
	P70	4
	P71	5
	P72	6
	P73	2
	P74	2
	P75	2
	P76	2
	P77	2
	P78	1
PHASE: Confirmatory data SOURCE: Non-FAIA affiliated organizations	P79	5	CEO (AI consulting firm), Chief Digital Officer, Data Scientist, Data Strategist, Development Director, Digital Engineering Lead, Head of AI Center of Excellence, Head of Digitalization, Professor of Computer Science, Project Manager (AI start-up), Senior Data Scientist, Senior Manager (AI Cloud Development), Staff Scientist
	P80	5
	P81	2
	P82	5
	P83	1
	P84	1
	P85	1
	P86	1
	P87	1
	P88	1
	P89	2
	P90	1
	P91	1

Source(s): Author's own creation/work

Note

1. Our use of “meta-learning” is distinct from the meaning of the term in the computer science field, where it refers to using machine-generated metadata to improve algorithms.