Implementation of large-scale health information systems

Purpose – The study aims to analyze the previous literature on government initiatives to implement health information systems (HISs). Design/methodology/approach – Proknow-C (Knowledge Development Process-Constructivist) was used in the selection of the literature and in the bibliometric and systematic analysis. Findings – The research identified a portfolio composed of 33 articles aligned with the research theme and with scientific recognition, as well as periodicals, authors, papers and keywords that stood out the most. Amongst the government initiatives in the 24 identified countries, England has been the most studied nation, and there is a certain prominence of research arising from developed countries. Electronic health records (EHRs) have been the most explored technology. Efficiency and safety of health care delivery, integration of information and among health organizations, cost reduction and economicity are the most expected benefits from government programs. The difficulties found are related to the broader context in which the system is inserted, to the management of the program, to technology itself and to individuals. The most emphasized difficulties identified in most countries were previous context marked by a lack of standardization/interoperability, acceptance of providers and users and project financing. The findings of the presentarticleprovideatheoreticalframeworkforfuturestudies,inadditiontoyieldingareplicableprocessforfutureuse. Originality/value – This research may be considered original as it analyzes – through a constructivism- structured process (Proknow-C) – the phenomenon under investigation by gathering bibliometric and systematic review data concomitantly. The countries and technologies reported emerge from the process itself. criteria: (1) time horizon: publications from 2008 to 2020; (2) search fields: titles, abstracts, (3) type of publication: scientific articles; (4) source of publication: scientific periodicals; and (5) publication language: English. The search activities were November 2020 and yielded a total of 7,265 articles, which correspond to the gross basis of articles.


Introduction
The world scenario, characterized by frequent changes in the social, economic, political and technological spheres, directs organizations to constantly reviewing their roles and processes. Without being an exception, institutions related with public health end up being deeply influenced by the environment and see themselves obliged to update their management policies and practices. The perspective of such institutions, which used to be devoted to assistance and devoid of the obligation to present management evidence, changed and they are currently on the path toward an entrepreneurial orientation, guided by performance, results and transparency (Lima, Schramm, Coeli & Silva, 2009;Botega, Andrade & Guedes, 2020).
The complexity of these organizations demands the generation, processing and availability of an expressive volume of information, which is essential for their processes and make up subsidies for decision-making in public health policies (Cresswell, Bates & Sheikh, 2013). These information include those referring to patients' health records (electronic health records), medical and nursing practices, management of clinical environments, knowledge and learning management, organization of administrative practices in the environment where the health care takes place, among others (Marin, 2010). By obtaining, processing and making them available for multiple stakeholders, an effective assessment is enabled not only in terms of administrative and operational efficiency, but of the greater purpose, i.e. the well-being of society (Morrison, Robertson, Cresswell, Crowe & Sheikh, 2011).
These initiatives are commonly characterized as complex programs or megaprojects (Price, Green & Suhomlinova, 2018), as they occur on a regional or national scale (largescale) and involve the coordination of multiple stakeholders, who usually have distinct interests and perspectives (Klecun, Zhou, Kankanhalli, Wee & Hibberd, 2019). Given the increasing frequency of implementation of large-scale HIS, researchers have reported the need to maximize knowledge and understanding of the perspectives utilized in such implementation (Cresswell & Sheikh, 2009;Cresswell, Williams & Sheikh, 2020b). A few scholars have been arguing (Ross, Stevenson, Lau & Murray, 2016;Sligo et al., 2017;Klecun et al., 2019) that there is a broad previous literature on punctual implementation of HIS and on local and organizational coverage of HIS. Research that addresses implementations with a broader comprehensiveness, i.e. large-scale, is still found, however in a smaller number, especially those of literature review, which is the case of the present proposal.
Therefore, this article aims to fill the literature gap identified, contributing to the selection and bibliometric and systematic analysis of a bibliographic portfolio about the phenomenon through a constructivism-structured process -Proknow-C (Knowledge Development Process-Constructivist) (Ensslin, Mussi, Chaves & Demetrio, 2015;Carvalho et al., 2020). As indicated by Carvalho et al. (2020), the selection of relevant literature through a structured process is a critical step in the development of systematic reviews.
The previous literature reviews on programs for implementation and assessment of largescale HIS usually aim to describe the program focusing on one or more specific countries (e.g. Ludwick & Doucette, 2009;Price, Green & Suhomlinova, 2018) and on specific health information technologies. In addition, these studies are not structured around the literature selection and bibliometric and systematic analysis in an integrated way through a bibliographic portfolio.
Thus, the present research provides a broader focus and contributes to the previous literature in three different ways: (1) by integrating concomitantly bibliometric and systematic review data of published literature, (2) by analyzing international initiatives of Large-scale health information systems implementation of large-scale health information technologies without predefining technologies or specific countries, which will emerge from the selection process itself and (3) by making use of a structured review process from a constructivist perspective (Proknow-C). Therefore, we intend to answer the following research issue: What is the state of art in the qualified literature about the implementation and assessment of large-scale HISs? More specifically, we intend to (1) select a bibliographic portfolio composed of a relevant, scientific bibliography aligned with the established theme; (2) to carry out the bibliometric analysis of this portfolio, exploring variables of the selected articles and their references; and (3) to carry out the systematic analysis of the portfolio by exploring the predominant countries identified in our research, the most studied technologies, implementation approaches, expected benefits and the difficulties faced by different countries.
This study may help and direct current and future investigation processes with the same nature. The research process and the main findings may be used by different researchers aiming to develop studies on the issue and may be used as a guide for practitioners, including those involved in initiatives of implementation of large-scale HISs.
In the upcoming section, HISs and the characteristics of their large-scale implementation will be described. In section 3, we will present the methodological procedures adopted, as well as the research framework. Subsequently, the application of Proknow-C and the outcomes of such method will be presented. Then, in the last section of this article, we present the conclusions and suggestions for future research.

Health information systems
The purpose of HIS is to produce and organize information and knowledge generated and used in the health care area in order to support the planning, the improvement and the decision-making process of the multiple actors involved in related processes (Lippeveld, Sauerborn & Bodart, 2000). The use of HIS can improve not only the health of individuals but also the performance of health care providers, providing increased levels of quality, financial efficiency and greater participation of patients in the care of their own health (Blumenthal & Tavenner, 2010). Several acronyms have emerged over time to designate electronic HISs, such as electronic health record (EHR), electronic medical records (EMR), health information exchange (HIE), computerized physicians order entry (CPOE), hospital information system and telemedicine/ telehealth/e-health (Blumenthal, DesRoches, Donelan, Rosenbaum & Ferris, 2006;Jha, Doolan, Grandt, Scott & Bates, 2008;Rosenthal, Seeman & Gibson, 2005;Lakbala & Dindarloo, 2014;Petroudi & Giannakakis, 2011;Spil, LeRouge, Trimmer & Wiggins, 2009).
Considering the national scope of implementation of such technologies, this sort of venture comprises the government definition of policies and standards that encourage the convergence of public and private interests in the development of an effectively functional national system (Coiera, 2009). Coiera (2009) still proposes a typology for regulatory models of implementation of large-scale HIS (top-down, middle-out, bottom-up), categorizing them according to the level of influence of government authority and the level of autonomy experienced by provider institutions in the process of developing, implementing and using the systems. While the top-down approach is characterized by a centralized management accomplished by the government, in the bottom-up approach the health care institutions make their own decisions about the system to be implemented, following the minimum interoperability standards. In an intermediate way, the middle-out approach combines elements of the other two approaches (Coiera, 2009. The implementation of electronic HISs at the national level has been considered a complex process full of challenges (Greenhalgh, Morris, Wyatt, Thomas & Gunning, 2013;Rippen, Pan, Russell, Byrne & Swift, 2013;, Cresswell et al., 2020a. Technical, human, social and organizational problems are frequently reported, which compromise the efficiency and effectiveness of these initiatives Greenhalgh et al., 2010;Murray et al., 2011;Peute, Aarts, Bakker & Jaspers, 2010;Spetz & Keane, 2009;Warth & Dyb, 2019), whichin turninvolve several interested parties necessary for the success of the implementation.
The actors involved in large-scale implementation programs comprise the government, health organizations, health professionals, patients, IT professionals and IT companies (McGinn et al., 2012;Sheikh et al., 2011;Rozenblum et al., 2011;Haried, Claybaugh & Dai, 2019). The wide range of interrelations among these actors generates a complex network of relationships, impairing the alignment of mutual interests (Greenhalgh et al., 2013;Greenhalgh et al., 2010;Robertson et al., 2010;Klecun et al., 2019;Cresswell et al., 2020). Amongst the several difficulties found in the implementation of large-scale HISs, the following stand out: (1) slower processes with a narrower scope than planned; (2) low degree of adaptability to local demands; (3) lack of alignment with work practices and users; and (4) negative impacts on the quality and effectiveness of patient care Murray et al., 2011;Peute et al., 2010;Spetz & Keane, 2009;Rippen et al., 2013;Robertson et al., 2010;Cresswell & Sheik, 2013).
Within this context, it is paramount that the implementations of HIS are assessed and that the relevant aspects of successful and unsuccessful initiatives are identified (Sligo et al., 2017). This article, through the methodological procedures described in the upcoming section, aims at selecting a bibliographic portfolio with scientific recognition that addresses global initiatives of implementation of large-scale HISs and at carrying out the analysis.

Methodological procedures
This study presents a literature review based on the constructivist perspective, utilizing a process known as Proknow-C (Ensslin et al., 2015). The first versions of Proknow-C appeared in 2007. Since then, it has been continuously improved and has aggregated features that explore the several possibilities of identification and analysis of bibliographic portfolios (Ensslin, Dezem, Dutra, Ensslin & Somensi, 2018). Proknow-C was chosen due to its constructivist orientation and its detailing and methodological rigor, in addition to its increasing applications (Ensslin et al., 2018;Carvalho et al., 2020) since it was developed. Every data collection and data analysis process of this research was guided by this instrument.

Proknow-C
Proknow-C consists of the following macro steps: (1) selection of a portfolio of scientific articles aligned with the research topic and with scientific relevance; (2) analysis of the portfolio through bibliometric criteria; (3) systematic analysis of the content of the portfolio from specific perspectives; and (4) definition of the research issue and aim of the scientific research (Ensslin, Ensslin, Imlau & Chaves, 2014).
The knowledge construction process begins with the selection of the bibliographic portfolio. This stage is intended to produce a research framework related to the phenomenon under investigation. The outcome of this stage is the set of articles (bibliographic portfolio) considered relevant by researchers and aligned with the research theme (Ensslin, Mussi, Chaves & Demetrio, 2015).
For the definition of the bibliographic portfolio, the starting point is the construction of a gross basis of articles. Initially, the research axes are defined, which represent the subthemes of the research. Through these axes, the parameters to be followed during the process of Large-scale health information systems structuring the theme are defined, represented by a set of keywords. Keywords are arranged through a Boolean expression, according to the research axes to which they are related. Subsequently, the databases where the search for scientific articles will be carried out are identified. The chosen databases must be aligned with the research theme, considering their relevance in the explored field of knowledge. Such representativeness can be evaluated through the number of articles resulting from the search through the defined Boolean expression. Additionally, filters are applied to the search to limit, for instance, a certain period and the types of document to be identified. The initial selection of articles must be validated by comparing the keywords used for the search with those presented in the articles found (adhesion test). If necessary, the search may be repeated including new research terms, which must be related to the research axes.
Then, the gross basis of articles goes through a refinement process with the application of some sequential filters: (1) elimination of redundant articles; (2) alignment of the title with the research theme; (3) scientific relevance, which corresponds to the number of citations of the article; (4) alignment of the abstract with the research theme; and (5) alignment of the full text with the research theme.
In the present research, in order to analyze the alignment of the articles with the research topic, the following inclusion criterion was defined: empirical or theoretical articles (i.e. that explore cases of the empirical literature) that portray, analyze or discuss programs of implementation and/or assessment of large-scale health information technologies carried out or supported by government initiatives. For these articles, the following exclusion criteria were defined: (1) studies focused on a single aspect of a HIS, such as usability; (2) planning, pre-implementation, or initial phase processes for the implementation of HISs; (3) assessment of the level of adoption of HIS in a given region and/or country; (4) eminently technical descriptions of HIS with little or no approach to implementation and/or assessment; and (5) studies focused on the description of approaches and/or methodological paradigms to investigate the implementation and/or assessment of HIS.
After establishing the bibliographic portfolio, the bibliometric analysis process begins (Van Raan, 2003), which consists in the survey and quantitative evidence of key information in the set of articles that make up the portfolio. In this stage, we analyze the distinctive variables of the selected articles as well as their references; more specifically, the scientific relevance of the periodicals, the relevance of the articles, the relevance of the authors and the most used keywords (Lacerda, Ensslin & Ensslin, 2014, Ensslin et al., 2015. The next step, known as systematic analysis, consists in analyzing the content of the portfolio based on the variables of interest to the researcher (Ensslin et al., 2018). The framework used in this research (Section 3.2) describes the key variables of the bibliometric and systematic analyses.
The described steps were carried out with the support of two different applications: EndNote X7.1 (Thomson Reuters, 2013) and Zotero 4.0 (Zotero, 2019). Endnote (Thomson Reuters, 2013) is a reference management software package; through its use, the identification of repeated articles in the portfolio was carried out, in addition to listing authors, utilized keywords and periodicals. This software was also used to make abstracts and full texts available for reading. Zotero 4.0 is a reference management software program to manage bibliographic data (Zotero, 2019); through its use, it was possible to identify the number of citations registered in Google Scholar not only for each article of the portfolio, but also for their references.

Research framework
The analysis variables are graphically represented in Figure 1, which illustrates the framework that guided the data analysis of this research. Based on the selection of a REGE 28,2 bibliographic portfolio representative of the research theme, the respective analyses were accomplished.
The bibliometric analysis involved: (1) the analysis of the relevance of the periodicals where the articles of the bibliometric portfolio were published, as well as the bibliographic references in each of the articles of this portfolio to identify and characterize those with greater relevance to the research theme; (2) analysis of the scientific recognition of the bibliographic portfolio articles and their references, considering their number of citations; (3) identification of authors and their relevance to the research topic of the portfolio and to the references; and (4) identification of the most frequent keywords.
The systematic analysis was carried out based on the analysis of the articles of the bibliographic portfolio considering the following variables: (1) countries where the initiatives for the implementation of the prevailing HIS took place; (2) the most studied health IT; (3) implementation approach adopted (top-down, bottom-up, middle-out); (4) expected benefits from national programs; and (5) main difficulties/barriers faced by the countries.

Application of Proknow-C and results
The steps of the Proknow-C application process referring to the selection of the bibliographic portfolio are detailed in Section 4.1 and presented in Figures 2 and 3. The bibliometric and systematic analysis are described in Section 4.2 and 4.3, respectively.
4.1 Selection of the bibliographic portfolio 4.1.1 Definition of research axes, keywords and Boolean equation. Three research axes representative of the theme analyzed were defined ( Figure 2). The first axis represents the analysis process, i.e. implementation and assessment. The second axis elucidates the object to be implemented or evaluated, i.e. HISs. The third and last axis indicates the scope of the process, i.e. large-scale.
For each established research axis, guiding keywords were defined ( Figure 2) based on the adhesion tests required by the method, gradually making them more aligned with the research theme. The combination of the three sets of keywords, representative of each of the three research axes, resulted in 144 combinations.
The Boolean equation that represents such combinations was applied to Scopus, IsiKnowledge, Ebsco, PubMed and ProQuest databases, limited according to the following (2) search fields: titles, abstracts, keywords; (3) type of publication: scientific articles; (4) source of publication: scientific periodicals; and (5) publication language: English. The search activities were carried out in November 2020 and yielded a total of 7,265 articles, which correspond to the gross basis of articles. 4.1.2 Filtering the gross basis of articles. The first action related to filtering the gross basis of articles was the identification and exclusion of 1,567 duplicated publications, which yielded a total of 5,698 articles to be analyzed according to the alignment of their titles to the research theme. In this step, the titles of the 5,698 articles were evaluated and, according to the inclusion/exclusion criteria (Section 3), 5,447 articles had to be eliminated. Thus, the basis of scientific articles, not repeated and aligned with the research theme through the titles, was composed of 251 articles. The procedure described so far is summarized in Figure 2.
In the next step, where the scientific relevance of the articles is evaluated, we verified the number of citations presented by each of the 251 articles according to Google Scholar. We determined a representativeness of 90% of the citations of these articles, which means that we only selected articles that had at least 40 citations. This set was composed of 57 articles and called Repository K (with title alignment and verified scientific recognition).
The other 194 articles of the portfolios with title alignment and not selected in this step compose the so-called Repository P (i.e. with title alignment but without verified scientific recognition). These articles were reassessed in a subsequent stage. In the next stage, the 57 articles with scientific recognition and aligned titles were evaluated in terms of alignment of their abstracts with the research theme. Again, the analysis was guided by inclusion and exclusion criteria (Section 3). Reading the abstracts of the 57 articles (Repository K) led to the exclusion of 33 articles. Thus, the databank of non-repeated articles with title and abstract aligned with the research theme and with verified scientific relevancecalled Repository Acomprised 24 articles. Additionally, we identified a set of 68 authors in the 24 selected articles, making up the author databank of Repository A.
At this point, the process retrieved the 194 articles from Repository P (with title alignment and without verified scientific recognition), which were submitted to a reassessment process. The aim of such reassessment was to identify eventual articles that could be incorporated into the bibliographic portfolio. The 194 articles were segregated according to the following filters: (1) articles published between 2019 and 2020; thus, for being recent, they still lack scientific recognition; and (2) articles published before 2016 by authors pertaining to the author databank of Repository A, i.e. authors who had already published papers with verified scientific recognition.
Hence, 71 articles were identified according to filter (1) and other 16 articles consistent with filter (2). The resulting 87 articles were submitted to the alignment criteria according to their abstracts; 14 aligned articles were identified, forming Repository B (non-repeated articles with aligned abstracts and potential scientific recognition). Then, they were incorporated into the articles of Repository A, constituting the so-called Repository C, with 38 non-repeated articles aligned with the research theme considering their titles and abstracts. Large-scale health information systems Subsequently, the 38 articles were fully read and five had to be eliminated for being misaligned according to the inclusion and exclusion criteria (Section 3). Thus, the primary portfolio was created, which comprises 33 non-repeated articles aligned with the research theme according to their titles, abstracts and full texts. Figure 3 illustrates the described procedures for the composition of the primary bibliographic portfolio.
The next step consisted of testing the representativeness of the primary bibliographic portfolio. The purpose is to identify if, within the references of the articles from this portfolio, there are other relevant articles with scientific recognition that should be incorporated into the portfolio itself.
4.1.3 Representativeness test of the primary portfolio. The representativeness test consists in listing all references from the primary portfolio's articles applying the same filters used to define the initial portfolio. The analysis of the 33 articles pertaining to such portfolio identified 2,020 references. Thus, 1,534 references had to be eliminated either because they were duplicated, because they did not correspond to the articles, or because they were published outside the time frame (2008 to 2020), which resulted in 486 articles. The analysis of the title excluded another 341 articles, while another 17despite being alignedwere already part of the primary portfolio, being also eliminated from this analysis. Therefore, the analysis yielded a total of 128 articles stemming from the references of the primary portfolio's articles, which were not duplicated, not part of the primary portfolio and aligned with the research theme according to their titles.
The 128 articles were then classified by the total number of citations on Google Scholar. We established a cut-off line representing the articles responsible for 80% of the citations. Then, the criteria of inclusion and exclusion described in Section 3.1 were applied to the abstract of the 29 resulting articles. In this assessment, however, we identified no article that could be incorporated into the primary bibliographic portfolio. The main exclusion factor was the fact that most of these articles were classified as theoretical and not directly related to empirical research. Thus, during this latest review, no article was added to the final bibliographic portfolio.
Hence, the selection of articles for the bibliographic portfolio identified 33 studies aligned with the research theme according to their titles, abstracts and full texts and with scientific recognition. These articles are listed in Table 1.
Afterward, we carried out the bibliometric analysis of the bibliographic portfolio and the articles in their references. Another relevant issue to be mentioned herein about the periodicals where the articles were published is the impact factor. In this research, we analyzed JCR -Journal Citation Reports (JCR) and Scientific Journal Rankings (SJR), verified through the databases Web of Science (JCR) and Scopus (SJR). Figure 5a presents the number of articles of each periodical of the portfolio, as well as its corresponding JCR impact factor. It is noticed that the periodical with the highest JCR impact factor is the British Medical Journal (BMJ) (JCR 5 30.31), which is the second journal with the highest number of articles of the portfolio (3 articles).
The      Large-scale health information systems The analysis of the prominent articles of the portfolio and references indicates an alignment with three different research axes: assessment and implementation; computerized health systems and large-scale processes. Figure 6b identifies the 20 most cited articles among the references of the bibliographic portfolio.
4.2.3 Identification of authors and their relevance to the research theme. In order to identify the most emphasized authors within the portfolio and considering the set of references, we verified the number of articles of each author in each of these two sets. The analysis of the authors responsible for the 33 articles of the portfolio, which is composed of 117 researchers, emphasizes A. Sheikh (participating in eight articles), K. Cresswell (responsible for seven articles) and A. Robertson, W. L. Currie and Z. Morrison (who participated in three articles each), conforming to Figure 7a.
We also identified the authors of the 128 articles pertaining to the references, which were aligned with the research topic. This databank of authors, composed of 346 researchers, highlights the participation of A. Sheikh in 12 articles, K. Cresswell in nine and T. Greenhalgh in six, which is illustrated in Figure 7b. Figure 7c allows the observation of the emphasized participation of some authors in the articles of the portfolio, as well as in the references of these articles. The upper quadrant on the right allows the identification of authors highlighted by the volume of articles not only in the portfolio, but also in the references.
Regarding the authors stressed in the bibliometric analysis, we highlight that Aziz Sheikh, who is responsible for eight articles in the portfolio and other 12 in the references, is professor and director of the Usher Institute, linked to the University of Edinburgh. His research interests comprise studies of asthma, health inequalities and use of health IT. Kathrin Cresswell, with seven papers in the portfolio and other nine articles in the references, is a psychologist and social scientist, who is also the director of innovation at the Usher Institute, linked with the University of Edinburgh. Operating in the field of information technologies for clinical studies, her main research interests are the use of health IT, sociotechnical analysis, qualitative methods, health innovation and information systems. Trisha Greenhalgh, highlighted for having two articles in the portfolio and six in the references, is a physician and professor of primary care health sciences at the University of Oxford. She dedicates to research related to the integration between social sciences and medicine, emphasizing health care organizations. Ashish K. Jha stood out for the citations of his work in the references of the bibliographic portfolio (five articles). He is professor of global health and director of Harvard Global Health Institute, linked with the Harvard School of Public Health, being active in health policies.
4.2.4 Keywords used in the bibliographic portfolio. The use of the EndNote X7.1 software (Thomson Reuters, 2013) allows the identification of 100 keywords utilized to characterize the articles of the bibliographic portfolio; the most frequent ones are presented in Figure 8.
Amongst the thirteen keywords defined based on the axes of the research, seven are to be found in this list. Such fact corroborates the alignment between keywords and the research topic, which validates its use in the search process. The most repeated keywords are implementation and EHR.

Systematic analysis
The aim of the systematic analysis was to highlight in the 33 articles of the portfolio the countries and technologies studied, the implementation approaches adopted, the expected benefits and the difficulties faced. We identified a set of 24 countries that were the focus of articles (Table 2) with a predominance of research in developed (18) in relation to developing countries (6).
The most widely studied national programs for the implementation of health IT are those in the United Kingdom, as they represent 22 articles of the portfolio. In this context, the British REGE 28,2 Large-scale health information systems initiative to implement a national system of EHRs linked to the national IT program (National Program for Information Technology -NpfIT) is the most approached research issue. Considered one of the most significant initiatives ever documented, it involved a total of 168 hospitals and 73 health care environments, corresponding an investment of £ 12.7 billion (Robertson et al., 2010). Such initiative, however, did not achieve the expected success, being discontinued in 2011 (Klecun, Zhou, Kankanhalli, Wee & Hibberd, 2019). Apart from the countries that make up the United Kingdom, the other most studied countries are the United States (eight articles) and Canada (seven articles). EHRs are the most studied technologies, which indicate the countries' search for integration and improvement of patient care. EHRs are defined as longitudinal medical records of patient health information, utilized as a basis for the integration of health care (Klecun, Zhou, Kankanhalli, Wee & Hibberd, 2019). These records follow pre-established patterns that take into consideration the possibility to share the patient history with other health organizations geographically dispersed, providing an integrated view of the patient's health (Klecun, Zhou, Kankanhalli, Wee & Hibberd, 2019;Ludwick & Doucette, 2009).
Improvements in the quality of health services are among the benefits expected by the national programs of the 24 countries identified in this research. Through the implementation of HISs, the countries expect improvements in the efficiency/agility of patient care, safety of health care delivery and patient's safety, privacy and security of health information, increased patients'degree of choice over their health care, improvements in the access of citizens to health services, integration and sharing of information between different health organizations; advances in the definition of health indicators and public policies, cost reduction, more consistent and efficient use of health resources and reduction of the gap between supply and demand of health care.
Regarding the adopted approaches to implement those systems, we identified the three approaches proposed by Coeira (2009) in the set of countries analyzed in this study. Coeira (2009) points out that the top-down approach used by several national initiatives to  2,3,4,6,9,11,13,14,16,17,18,21,23,24,26,27,28,29,30,31   implement HISs may be the main reason of their frequent failures. Among the problems identified, one highlights the fact that a unique, standardized and imposed (by the government) solution does not apply to the variety of local problems and asymmetries. Thus, the author defends the use of the middle-out approach as the most appropriate way for governments to achieve goals related to national policies, preserving the interests of involved institutions and providing solutions to local asymmetries.
In line with such perspective, the outcomes of this study show that the adopted implementation approach is among the difficulties identified in the different countries. In the case of England, for instance, previous research (Robertson et al., 2010 indicated that one of the causes of failure of the national IT program (NPfIT) lies in the topdown approach, which generated delays in the process and user frustrations. In the case of the United States, the bottom-up approach was unable to stimulate the adoption of technology by large providers (Wilson & Khansa, 2018). Failures resulting from the approaches utilized seem to have affected the future directions of some countries. England and the United States, for example, have been directed more efforts toward the implementation of a middle-out approach (Fragidis & Chatzoglou, 2018, Price, Green & Suhomlinova, 2018. In addition to the implementation approach adopted, it was evident that all countriesregardless of their level of developmentfaced a set of difficulties or barriers when implementing health information technologies nationally, which strengths the dimension and complexity of this sort of program. Of the articles analyzed, we identified reports of 34 main difficulties (Table 3).
When assessing the set of difficulties faced by the countries, the ones that stood out in more than half of this set are the previous context of lack of standardization and interoperability and the consequent difficulty in integrating systems, regions and health suppliers; resistance, skepticism, acceptance of health care providers and users; and project financing and budget constraints. It is important to highlight herein that most part of those difficulties are related to the management capacity of the programswhether at the government or at the organizational leveland to the conciliation of the different needs of the several stakeholders involved.
In general, the difficulties identified are related to a wider context of system implementation (e.g. instabilities in the country's political and economic contexts, changes in market structures, suppliers and technologies), to the way the program is managed and conducted (e.g. project management, change management, implementation approach adopted), to technology itself (e.g. usability, performance, alignment with work practices) and to individuals (e.g. resistance, skepticism, acceptance of the system, previous experience with computers).
These findings corroborate other studies (Cresswell, Williams & Sheikh, 2020a, Greenhalgh et al., 2010 that emphasize that this sort of initiative involves a set of social, organizational and human transformations that must be considered to ensure that the benefits expected by the different stakeholders are, in fact, achieved. As pointed out by Price, Green & Suhomlinova (2018), national, effective IT systems require the ability to understand and manage complex and dynamic relationships of the several organizations involved.

Final considerations
The purpose of this article was to carry out the selection, bibliometric and systematic analysis of a bibliographic portfolio about the topic "implementation and assessment of large-scale health information systems" through a structured process (Proknow-C). The selection stagebased on a set of established criteriaidentified a databank of 33 articles aligned with such research topic.  The bibliometric analysis allowed the identification of the periodicals International Journal of Medical Informatics, Journal of the American Medical Informatics Association (JAMIA), Medical Informatics and Decision-Making (BMC) and British Medical Journal (BMJ) as those with more publications about the research theme. These journals stood out for the expressive volume of articles in the bibliographic portfolio and in the references.
In the author databank identified in the portfolio and in the references, two authors stood out: Aziz Sheik, PhD (eight articles in the portfolio and 12 articles in the references) and Kathrin Cresswell, PhD (seven articles in the portfolio and nine in the references). Both are researchers at the Usher Institute, linked to the Edinburgh University.
In the selected portfolio, the ranking of academic relevance ( The bibliometric analysis allowed for the identification of the most frequently used keywords in the selected articles, namely: Electronic Health Records and Implementation. In addition, the set of keywords identified in the articles corroborates the adhesion of the keywords initially used in the search process.
The systematic analysis indicated a set of 24 countries that were the focus of national initiatives for the implementation of health ITit is observed, however, a prominence of studies carried out in developed countries. England is the mostly studied country. The most discussed technology identified in the articles is the EHR, which indicates that countries are aiming to provide patients with an integrated health care system. We observed a shift in the orientation of some countries, which initially used to adopt top-down and bottom-up implementation strategies, toward the middle-out approach.
Through the implementation programs of HISs, the different countries intend to improve the efficiency and safety of health care delivery, to integrate information and health organizations, to reduce costs and to provide resource savings. Improvements in service quality to citizens are expected by all countries.
In the pursuit of these benefits, national governments have been facing several difficulties. From the analyzed articles, 34 main difficulties emerged, which are related to a broader context of system implementation, to the management of the program, to individuals and technology itself. The difficulties related to resistance and acceptance of health organization and users; to the previous context of lack of standardization and interoperability; and to project financing and budget constraints were the most frequent ones identified in most part of the countries.
In general, the difficulties emphasize the complexity of implementing national systems of health IT; in some cases, they are underestimated by countries and end up compromising the success of these initiatives and the pursuit of expected benefits. An effective management of the program focused on integration and on a shared vision between the diversity and heterogeneity of those involved stands out as a basic requirement.
Among the theoretical implications, the results presented herein contribute to the creation of an updated and systematically constructed theoretical framework on the research topic. The findings add knowledge to literature on implementation and assessment of large-scale health IT, as well as on management of complex programs and projects. Additionally, the study also provides a replicable process, which can be used in its own update.
In portraying government initiatives from different countries, this research adds to managerial implications for government agencies and health institutions. The benefits expected from these initiatives are achieved through the effective management of the difficulties identified in the reported experiences. These difficulties may be translated into Large-scale health information systems lessons to be learned for those involved in this sort of government program. Thus, the effective management of these programs has a direct impact on the quality of health care service and individuals' well-being. Ultimately, this research has implications for public policies, indicating the need for the development of health national policies and e-health, which must support and be aligned with the implementation of large-scale health IT in order to enhance the expected benefits and minimize the barriers inherent in this kind of initiative. These benefits make a difference in the quality of the health care service provided to citizens and can contribute to atypical contexts, such as what is being experienced with the pandemic caused by COVID-19. These contexts require efficiency and safety of health care delivery, reliable and fast information and the integration of systems and among health institutions.
We reaffirm that the aim of this study is not to create a theoretical framework per se, but to contribute to the creation of knowledge through the provision of a theoretical basis about the research topic, which was elaborated through a rigorous, structured and systematic process. Despite following a structured process, the selection of articles may also be influenced by a few subjective factors inherent in researchers, which already indicates one of the limitations of this study. Another limiting factor is that the main difficulties faced by the countries when implementing their programs refer to the selected bibliographic portfolio. Thus, the fact that other difficulties have not been identified in this set of countries does not mean that they were not faced by them.
Based on the findings of this study, we identified a few suggestions for future research, namely: (1) the study of initiatives to implement HISs in developing countries in order to expand knowledge about these countries; (2) comparative studies between developing and developed countries; (3) analysis of strategies or actions adopted by countries to minimize the difficulties identified in this study; and (4) update or periodic replication of the process carried out by us to maintain a timely analysis on the subject.