Validation of a multiple criteria tool for healthcare facilities quality evaluation

Purpose – Several healthcare quality assessment tools measure the processes and outcomes of the care system. The actual physical infrastructure (buildings and organizational) aspects are, however, rarely considered. The purpose of this paper is to describe the process of validation and weighting of an evidence- informed framework for the quality assessment of hospital facilities from social, environmental and organizationalperspectives tocomplement otherassessments. Design/methodology/approach – Sustainable High-quality Healthcare version 2 (SustHealth v2) is the updated version of an existing framework composed of three domains (social, environmental and organizational quality). To validate and establish a relevant weighting, interviews were conducted with 15 professionals within the ﬁ eld of healthcare planning, design, research and management. The study has been conducted through semi-structured interviews and the application of the Simon Roy Figueras (SRF) procedure for the elicitation of weights criteria. The data collected have been processed through the DecSpace web platform. Findings – Among the three domains, the organizational qualities appear to be the most important (W = 49%), followed by the environmental (W = 29%) and social aspects (W = 22%). Relevant indicators such as future-proo ﬁ ng, way ﬁ nding and users ’ space control emerged as the most important within each macro-area. Those results are con ﬁ rmed by the outcome of the interviews that highlight user/patient-centeredness, way ﬁ nding strategies and space functionality as the most important concepts to foster in existing healthcare facilities improvement. Practical implications – The study highlights important structural and organizational aspects that hospitalmanagers andplanners can considerwhen dealing with healthcare facilities ’ qualityimprovement.


Introduction
Hospital facilities reflect the complexity of healthcare systems and host diverse and multiple daily users, integrate advanced technologies and systems and have a public role as health promoters. Hospitals are generally huge and complex institutions that are constantly in transformation both from the organizational and from the physical point of view in terms of sustainability, space management and facility management issues (Capolongo et al., 2016;Jiang and Verderber, 2017;Prugsiganont and Jensen, 2019;Talib et al., 2013;Yousefli et al., 2017). The built environment, architecture for healthcare, is considered by a growing amount of research to have an important part in contributing to high-quality health services (McKee et al., 2002). Nevertheless, design quality and effects on health outcomes have been described as difficult to define and evaluate precisely and been widely debated in terms of measurability (Anåker et al., 2017).
At the same time, in the field of medicine and healthcare management, measure-oriented approaches and quality assessment tools are regularly used as the foundation of clinical activities to foster the quality improvement of the services delivered.
Based on Donabedian's model, several quality assessment tools are available for process and outcome domains while the infrastructure aspects (buildings and organizational settings) of the healthcare system are often not considered (Zengul and O'Connor, 2013).
In the past two decades, growing attention and awareness have emerged considering the topic of built environment evaluation and, in particular, on environmental sustainability with several building performance evaluation (BPE) tools developed and currently in use. Those instruments are able to assess different design stages, among which also the operating phase with the use of post-occupancy evaluation (POE) strategies . Some versions of tools such as Leadership in Energy and Environmental Design (LEED) are also available with a healthcare version but they are linked to certification bodies, prescriptive evaluations and technical requirements that might vary region by region and it is difficult to extract strategic improvement directions out of these. Thus, despite the growing attention on the topic, there is a lack of specific tools for healthcare facilities that enable structured descriptions for analysis or that enable evaluation in a complete way analyzing set performance parameters and contributing to establishing systematic studies of healthcare facilities.
Therefore, to bridge this gap, the aim of this paper is to shed light on the definition of an evidence-informed framework describing the methodological process of validation and weights assignment that contributes to this development.

Research objective
The purpose of this paper is to contribute to the development of evaluation approaches for healthcare facilities that enable feedback to core activities, design, building and maintenance processes for effective production and use. While the overarching objective is to contribute to systematic studies of healthcare facilities, the actual contribution aims at describing the process of validation and weighting of the evidence-informed framework sustainable high-quality healthcare version 2 (SustHealth v2), as a quality assessment tool for hospital facilities evaluation from the social, environmental and organizational perspective. The present research addresses existing gaps in the field of healthcare built environment evaluation.

The need for evaluation
The investments in healthcare facilities are small compared to the running costs of the healthcare system using them. Still, for the investors, public and private and the healthcare organizations, they represent a project with apparent costs that are feasible to grasp, have opinions about and relate to their own activities. Management and staff have needs and expectations and the clients have an objective they want to achieve. In a way basic, but as health-care is part of our society it is also related to a series of interests/stakeholders ranging from politics to users. Applicable systematic evaluation methods and approaches have a role to fill in this area. Not least for transparency, decision-making support and evaluation of alternatives.
Indeed, hospitals need to keep functioning as efficient production facilities while, at the same time, incorporating patient-centered care, user centeredness and being open and welcoming. The investments required in building or for renovating a hospital are usually also so large that the organization must base that investment, along with its programming and design decisions, on rigorous analysis of long terms consequences. As stated by McKee et al. "the first step in achieving the desired outcome of high-quality, cost-effective care is ensuring that the right physical structures are in place" (McKee et al., 2002). Within such complex system, it is very important to assess the impact of design solutions on the efficiency of medical processes, expanding the scope of design work to functional planning including features such as logistics, public space, wayfinding, layout, ergonomics, organization, infrastructure (Wagenaar and Mens, 2018). To do so, hospital organizations usually use external consultants, professionals and companies to check the architectural and organizational functionality and ask for suggestions for improving the existing facilities or for designing new buildings. Those decisions are usually based on experiences and best practices but rarely on systematic collection of evidence due to the lack of usable and effective assessment tools.
There is a need for a framework, a collection of elements grouped in areas, criteria or indicators that define which are the content of the evaluation. The assessment tool is the operational instrument that enables the collection of data and it can consist of checklists, surveys to the users, documentation analysis, etc. (Dell'Ovo et al., 2018).

Assessment tools construction
Generally, a tool for assessment of the built environment is based on a hierarchical structure and the different parts can be related to a decision tree. As shown in Figure 1, the higher part is formed by fundamental and interconnected macro-areas . Each area is further divided into a hierarchical framework of criteria and indicators (C&I type), which are the elements concurring to the final score of each specific aspect. Each criterion relates to one key macro-area and may be described by one or more indicators. Eventually, an indicator might be composed of one or more very specific items to measure or verify with specific rationale and methods. According to the definition of the standard UNI 11097, an indicator is: The information, qualitative or quantitative, that is able to evaluate its change during the time and to verify the defined quality goals, to take the correct decisions and choices (UNI 11097:2003(UNI 11097: , 2003. Indicators can be either qualitative or quantitative and not only allow to compare different situations but they can also give insights over time because of the indicators' periodical measurement. In particular composite indicators can summarize complex, multi-dimensional realities with a view to supporting decision-makers (DMs) (European Commission, Organisation for Economic Co-operation and Development and SourceOECD (Online service), 2008).
Indeed, this hierarchical structure reflects the complexity of decision-making processes and the possibility of defining an objective system of measurement and comparison between different alternatives (Keeney, 2013;Podvezko, 2011). In some cases, indicators can be considered also as pre-requirements for accessing the evaluation. Such frameworks are widely used by many government agencies, non-governmental organizations and academic researchers to define sustainability monitoring and evaluating programs. The macro-areas might have different relevance in the evaluation system according to the importance and the impact they hold on the sustainability of an operative healthcare structure.

Building performance and post-occupancy evaluations
A very effective and well-structured approach in this direction is the POE, defined as the process of systematically comparing actual building performance after completion and occupation (Connellan et al., 2013;Federal Facilities Council, 2002;Preiser, 1989Preiser, , 2003. Starting from the 1970s, a number of discussions concerning the evaluation of the built environment have been recurring with related methods development (Markus, 2001). With the recent birth of the green building concept, this approach also incorporated the BPE methodologies along with the spreading attention to environmental sustainability and ecology (Chew et al., 2017;Li et al., 2018;Meir et al., 2009;Preiser et al., 2018). This has also led to the development and international diffusion of tools such as LEED, building research establishment environmental assessment method (BREEAM) and many others. Specific facilities require dedicated assessment tools (Abisuga et al., 2019). Some of the mentioned instruments tried to develop a health-care-related version such as LEED healthcare, but they are not updated and focus mainly on ecological sustainability parameters. Indeed, existing BPE and POE instruments are not adequate to fully appraise hospital characteristics and features and unable to set benchmarks for their evaluation. Moreover, existing research on the topic is scant and the potentialities of integrating theory with practice are still to be unfolded.

Susthealth tool
One example of a POE assessment tool specifically designed for hospitals is sustainable high-quality healthcare version 1 (SustHealth v1) which is an Italian qualitative assessment tool, developed from a collaboration between Politecnico di Milano and Politecnico di Torino . The evaluation framework is composed of 3 macro-areas, 13 criteria, 70 indicators weighted and aggregated with the analytic network process (ANP), a generalization of the AHP multicriteria methodology. The tool is not easy to apply due to the Multiple criteria tool different scoring systems and the need for involving several users and therefore has been applied in just two cases. Additionally, the tool is dated back to 2009 and, as then new priorities emerged in healthcare planning, as well as more appropriate weighting methodologies for the specific complex topic of healthcare facilities.
Starting from previously published literature and assessment tool reviews, a new version is under development named SustHealth v2. This updated version is composed of 3 macroareas, 17 criteria, 52 indicators operationalized with 209 variables in the form of a checklist. This can secure more usability and velocity in the application and provide a more precise picture of the whole facility in less time.

Methodology
To make the framework more reliable, before testing the tool in an actual case study, a validation and weighting phase has been conducted. This is a delicate but decisive phase for the success of the analyzes and includes the attribution of importance or weight, to each of the criteria so that the hierarchies that mutually regulate them are defined.
The weighting procedure further described in this paper has been carried on with the support of 15 figures experienced and familiar within the field of healthcare planning, design, research and management in Swedish and Italian healthcare contexts. The study has been conducted through semi-structured interviews by applying the Simon Roy Figueras (SRF) methodology (Figueira and Roy, 2002) and data collected have been processed through the DecSpace software. The results have been normalized, aggregated and compared with the interview outcomes.

Multiple criteria decision aiding and Simon Roy Figueras methodology
The complex nature of the healthcare environments, the multitude of stakeholders and the presence of both qualitative and quantitative indicators, led to the selection of an assessment and weighting system from the Multiple Criteria Decision Aiding (MCDA) family. MCDA methods are often used to make a comparative evaluation of alternate projects assessed considering heterogeneous measures (Köksalan et al., 2013) and aim at supporting DM with elements to reflect, conjecture, discuss and argue about decisions in which a plurality of points of view are taken into consideration (Greco et al., 2016). Those methods can be chosen according to the nature of the information (quantitative or qualitative/mixed methods), the level of compensation allowed (compensatory, partially or non-compensatory methods) or other characteristics of the decision problem to face (Ferretti and Montibeller, 2016;Siskos and Tsotsolas, 2015). Within this context, the weights assignments phase needs to be further analyzed given its important contribution in influencing the final result and given the presence of multiple and sometimes conflictual opinions to consider. Among the several methods developed within the MCDA family, an approach that has gained more and more attention for helping in eliciting parameters is the SRF method (Figueira and Roy, 2002). This revised version of Simon's MCDA methodology exploits the deck of card method (DCM) which permits the DM to express differences in attractiveness by adding cards between consecutive elements (Corrente et al., 2019;Figueira and Roy, 2002). The main innovation in this approach consists of associating a "playing card" to each criterion, DCM and the interviewee is asked to rank them considering their importance in achieving the final objective. The distance and consequently the importance between each criterion can be further increased by introducing blank cards which are provided in a second moment (Corrente et al., 2019;Figueira and Roy, 2002). The phases developed by the interviewee, starting from the ranking to the use of blank cards, if needed, allows a rather intuitive understanding of the aim of this procedure given the practical experience. This approach F 39,5/6 compared to the more traditional ones (i.e. AHP and ANP) appears to be more intuitive for the DMs, as it does not require to assign numerical values to the different variables but order preferences. The last step of interaction concerns the elicitation of new additional information from the DM which changes certain computing rules of the former method developed by Simon. The relationship between the first and the last criterion of the ranking is asked by defining how many times the first is more important than the last (Dell'Ovo et al., 2020).
Indeed, the purpose of this method is to allow the DM to express merit judgments on the purposes of the analysis and to reason about the hierarchies that are established between the criteria.
Starting from the selection of a consistent group of experts to interview, passing through the three interactive steps previously described until the weights visualization, six different phases can be recognized, namely, expert selection; card design; interaction phase; card ranking phase; ratio identification phase; computation phase. These propaedeutic phases are going to be further described below.
4.1.1 Expert selection. The stakeholders' analysis has been performed to identify both experts and DMs involved in the decision process. In fact, while the experts can contribute to influencing the final decision given by their specific competences in the field under analysis, who is in charge to make the final choice is represented by the DM. Given this definition formulated by Dente (Dente, 2014), DMs have been asked to evaluate and weight the macroarea level while experts have been dedicated to criteria.
In detail, the DM has proven knowledge and skills in the disciplinary field of the subjects being evaluated, able to understand the priorities that exist in the choice and the relationships between the various parameters. The figure of the DM has been chosen from the Italian context and calibrated in relation to the area in which the study fits so as to make the weighing of the criteria as reliable as possible.
The experts have been selected for their role, achievements or recognized experiences in the field of hospital planning, design, assessment both from academia and practice environment in the Swedish context. Although the famous pyramid in the realm of Evidence Based Medicine (EBM) that propose a hierarchy of knowledge quality or reliability place the expert opinion as the bottom place highlighting the role of empirical evidence, authors argue the need of rediscovering the role of "traditional" authority (Guyatt, 1992;Tonelli, 1999). Reliance on expert opinion is indeed often a necessary requirement for unfolding complex topics. Indeed, if applied to complex systems and decision-making challenges related to the built environment, the role of experts is instead very relevant. Especially given the distribution of responsibilities driven by an increasing number of professional fields in design and construction.
The selected stakeholders have been clustered according to their main areas of interest such as; environment, social or organizational qualities (Tables 1 and 2), have been contacted and engaged with a preliminary description of the project. After their acceptance, a formal meeting has been scheduled and conducted in person or through the use of videoconference or telephone calls.
4.1.2 Cards design. The SRF procedure has been carried on through the DCM which concerns the physical representation of the criteria to weight with cards. Therefore, a specific set of cards have been designed containing information about the name of the criterion or macro-area represented and, if necessary, some additional details (brief description, case study, criterion label, notation, etc.). Moreover, a representative icon has been added to each card to increase understanding of the meaning of the criterion. Multiple criteria tool 4.1.3 Interaction phase. This phase has been conducted in an informal environment as a one-to-one dialogue between September 2019 and January 2020 in English or Italian language (considering the nationality of the interviewee) and each session lasted about 30 min on average. The interviews have been recorded and pictures have been taken under the explicit consent of the interviewee.
The weighting session has been anticipated by a moment of semi-structured interviews where each expert has been asked to give their personal and professional opinion on specific topics. The questions have been formulated as follows: If you had a maximum of three concepts, which one would you use for defining quality in the healthcare built environment? Assuming you have the possibility of improving your hospital which aspects would you look at?
This qualitative assessment has been very useful for the collection of insightful strategies, for the preliminary investigation and confirmation of the need and gaps identified and for the following phase of weighting.
4.1.4 Card ranking phase. A deck of cards representing each criterion has been provided to each expert to proceed with the weights elicitation. Following the SRF procedure, experts have been asked to rank the cards from the most to the least important. If two or more criteria had the same importance (i.e. the same weight) the respective cards could be placed on the same level. Furthermore, a set of blank cards have been provided and they could be added between two criteria to increase their distance. This step can be repeated until a À PhD, professor nursing in a school of health and social studies À PhD, a visiting researcher in healthcare and member of municipality planning À PhD, guest professor and worldwide renowned expert in EBD À Partner of architectural office and artistic professor À Healthcare development director and partner of architectural office and adjunct professor À Director healthcare division and Partner of an architectural office À Architect at the architectural office and adjunct professor À Director of a research center for healthcare improvements with a strong background in healthcare organization À MD, PhD, unit manager for the program office at the regional university hospital À Head estate manager at the regional level À Head of investment planning and property management at the regional level correct and satisfactory visualization is achieved. The greater the distance between the mentioned weights of the criteria, the greater the number of blank cards. No blank cards mean one unit (u) for measuring the intervals between the two weights, one white card means two units (2 u), two cards three times (3 u), etc. (Figure 2). 4.1.5 Ratio identification phase. The last step introduced by Figueira and Roy, with respect to the Simos method, consists of asking the stakeholder to highlight the ratio (value "z") between the weight of the most and the least important criterion considering a range between 0 and 100 (Figueira and Roy, 2002;Siskos and Tsotsolas, 2015).
4.1.6 Computation phase. To perform the computations, the web-based platform DecSpace, which makes use of MCDA methods, has been applied which allows users to elaborate inputs previously detected by the interviews and to obtain a normalized set of weights. To result with a final and univocal weighing, the outcome elaborated for each expert has been further processed and, assuming that they all have the same importance, the arithmetic mean has been calculated and chosen as aggregation procedure (Figure 3).  Multiple criteria tool 5. Discussion of the results 5.1 Quantitative results emerged from the weighting procedure First, the assigned weight of each macro area was calculated. After the normalization procedure, the final ratios are reported in Table 3 both for macro-areas (W) and criteria within its macro-area (w).

Qualitative results emerged from the semi-structured interviews
Semi-structured interviews highlighted both transversal features, that could be applied to each area and specific vertical aspects to attend and consider. In particular qualitative data have been collected in the form of expert opinions and descriptions of experiences. According to their respective area of expertise, the interviewees were able to highlight features relevant for healthcare facilities that can be addressed during the process of healthcare facility quality improvement from an organizational and physical point of view.
The different concepts are highlighted in Table 4.

Transversal issues throughout the design process
The concepts collected during the interviews can indeed be related to some transversal phases such as the design process, built environment characteristics and facility operation and management.
In the design process improvement area, the concept of user-centeredness has been mentioned from many perspectives by several experts. Customer-centeredness, patientdriven or patient-centeredness were the concepts that have been addressed by most of the experts from multiple areas. This idea is in line with the social role that healthcare facilities have. Indeed, throughout the design process, it is important to match the final design choices with the healthcare facility's purposes, values and vision.
In the built environment characteristic area different features have been highlighted such as localization, safety, usability, logistics, functionality and flexibility or future-proofing. Additional attention has been dedicated to all the features that a built environment could have to improve patient and staff well-being including perceptive quality improvement through soft-qualities and wayfinding features. Therefore, the concept of a healing environment and hygienic aspects emerged as also important. Finally, features in the Facility operation and management area have been clustered using concepts that emerged from the experts mainly related to the improvement of some characteristics within the operative life of the healthcare buildings. Environmental sustainability in terms of energy efficiency and reduction of energy consumption has often Multiple criteria tool been mentioned as one of the most important criteria that healthcare facilities should meet. Efficiency, effectiveness and operational cost control appear also as significantly important.

Main features highlighted
The two typologies of data collected (quantitative and qualitative) are significantly coherent and some common features can be highlighted.
First, facilities with future-proofing characteristics, both in terms of space and organizational flexibility, appear to be very much suggested by the stakeholders. Although the concept of flexibility is highly complex and include a variety of features such as transformability or scalability, the idea of designing a resilient facility able to adapt to the unexpected epidemiological, technological, social changes is always desirable, especially in the light of the recent COVID-19 pandemic (Capolongo et al., 2020). An up-to-date facility that is able to transform according to the evolving internal or external need should be addressed, especially considering that, as one interviewee stated "[. . .] we cannot refurbish a building every time we change the use. We need a 'general' building where you can have all the possible treatments in the same area [. . .]." Although attention to staff well-being is emerging more and more, patient safety appears to be one of the most important features highlighted both in the weighting phase and in the interviews. The concept of patient safety is also connected to the idea of a patient (user)centeredness and the built environment can have an important yet underrated impact on this. For example, studies highlight the impact of single-patient rooms on health-careassociated infections or the role of surfaces/materials and spatial layout on patient falls (O'Neill et al., 2018).
The third topic mostly mentioned and highly ranked is the wayfinding concept. As mentioned by interviewees, one of the objectives of the design is that "[. . .] the environment says 'welcome to me, I want to take care of you' [. . .]" to the patients, staff and visitors. This very subjective result could be achieved through rigorous planning of the logistics, functional and layout features (flows and department connections) along with synergies based on signage and furniture design.

Conclusions
The concept of quality assessment of methods and tools is challenging and complex especially when applied to healthcare buildings like hospitals. Hospitals are complex infrastructures that represent articulated institutions and integrated service deliveries. Existing BPE and POE tools are not specific enough and the new version of the tool presented in the paper is a possible solution to bridge this gap by incorporating evidenceinformed indicators and systematic weighting methodologies. The approach chosen with a weighted set of criteria and indicators was found to support the evaluation process.
In particular, among the three domains, the organizational qualities appear to be the most important, followed by the environmental and social aspects. Relevant indicators such as future-proofing, wayfinding and users' space control emerged to be the most important within each macro area.
Those results are confirmed by the outcome of the interviews where user centeredness, wayfinding strategies and space functionality are highlighted as the most important concepts to pursue when dealing with existing healthcare facilities improvement.
The SRF methodology adopted guarantees reliability of the weights assigned to each criterion and macro areas contributing to the field of healthcare building research and evaluation by providing a weighted instrument to be tested in further studies.

Research limitations and implications
The tool is based on Italian and Swedish experts' contributions. This has given interesting and internally valid data, but to validate it further it should be tested in case studies, as well as discussed in the wider FM community. The validated assessment tool presented, although it needs to be tested for usability check, present a significant advancement in the research field of built environment evaluation and healthcare facilities as it merges an evidence-informed approach and a systematic weight assignment procedure.

Practical implications
The tool can be used in practice as a decision support instrument on several levels in the management of healthcare facilities or investment in new projects. On a strategic level, to evaluate a number of facilities or projects, create systematically based data and thereby supporting long term planning. On the tactical level, it can be used to plan and prioritize investments in new projects or existing building stocks. On the project level, it can be used to guide decision-making on what focus has, choose or prioritize.

Originality/value
It is the first time that such a tool with a focus on hospital built environment is validated supported by multicriteria analysis methodologies and with a wide panel of experts in a European context.

Further research
The weighted framework will be tested in actual hospital case studies with the support of technical, sanitary and FM departments.