Applying the Delphi method to determine best practices for outsourcing logistics in disaster relief

Timo Gossler (Research Institute for Supply Chain Management, WU (Vienna University of Economics and Business), Vienna, Austria)
Ioanna Falagara Sigala (Research Institute for Supply Chain Management, WU (Vienna University of Economics and Business), Vienna, Austria)
Tina Wakolbinger (Research Institute for Supply Chain Management, WU (Vienna University of Economics and Business), Vienna, Austria)
Renate Buber (Institute for Nonprofit Management, WU (Vienna University of Economics and Business), Vienna, Austria)

Journal of Humanitarian Logistics and Supply Chain Management

ISSN: 2042-6747

Article publication date: 5 December 2019

Issue publication date: 5 December 2019




The purpose of this paper is to determine best practices of aid agencies for outsourcing logistics to commercial logistics service providers (LSPs) in disaster relief. Moreover, it evaluates the application of the Delphi method for research in humanitarian logistics.


The paper is based on a two-round Delphi study with 31 experts from aid agencies and a complementary full-day focus group with 12 experts from aid agencies and LSPs.


The study revealed 12 best practices for outsourcing logistics in disaster relief and a compilation of more than 100 activities for putting these practices into action. Experts consider a proper balance between efficiency and compliance, a detailed contract and a detailed service request most important. Additionally, the Delphi method was found to be a promising technique for research on humanitarian logistics.

Research limitations/implications

By critically examining the Delphi method, this study establishes the basis for a wider application of the technique in the field of humanitarian logistics. Furthermore, it can help to prioritize future research as the ranking of practices reflects the priorities of practitioners.

Practical implications

The paper provides guidance to practitioners at aid agencies in charge of outsourcing logistics.


This research is one of the first in the field of humanitarian logistics to apply the Delphi method. Moreover, it addresses the lack of literature dealing with approaches for building successful cross-sectoral partnerships.



Gossler, T., Falagara Sigala, I., Wakolbinger, T. and Buber, R. (2019), "Applying the Delphi method to determine best practices for outsourcing logistics in disaster relief", Journal of Humanitarian Logistics and Supply Chain Management, Vol. 9 No. 3, pp. 438-474.



Emerald Publishing Limited

Copyright © 2019, Timo Gossler, Ioanna Falagara Sigala, Tina Wakolbinger and Renate Buber


Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at

1. Introduction

Outsourcing logistics in disaster relief operations provides considerable benefits to aid agencies, such as cost efficiency, flexibility and scalability (Baharmand et al., 2017; Cozzolino et al., 2017; Thomas and Fritz, 2006). Therefore, many aid agencies make strong use of outsourcing, and multiple billion US$ are spent every year by aid agencies on logistics services (Binder and Witte, 2007; L’Hermitte et al., 2016). Commercial logistics service providers (LSPs) are an integral part of any disaster relief operation, both at the international and the local level (Sánchez Gil and McNeil, 2015; Vega and Roussat, 2015).

However, many organizations fail to build successful collaborative relationships with LSPs (Bealt et al., 2016). Often, they are dissatisfied both with the perceived performance and the related costs (Cottam et al., 2004; Schulz, 2009). The success of outsourcing engagements is affected by specific challenges of the humanitarian environment. For example, aid agencies’ high staff turnover and unpredictable funding cause issues for building relationships with LSPs (Thomas, 2003; Van Wassenhove, 2006). Likewise, different cultures and working styles cause frictions during the collaboration (Nurmala et al., 2017). In the light of these challenges, aid agencies seek guidance on how to best leverage the advantages of outsourcing while reducing the risks and downsides of such engagements. Their scope of action is wide in this regard, covering activities such as provider selection, contract negotiation, performance evaluation or supplier management (Bagchi and Virum, 1996; Gould, 2003; Sink and Langley, 1997). Some approaches in these areas, called best practices, consistently show results superior to those achieved by other approaches (Business Dictionary, 2018; Supply Chain Council, 2008). Taking a more strategic and structured approach to outsourcing by replicating such best practices can help aid agencies to be more successful in outsourcing engagements (Bealt et al., 2016). Therefore, in response to aid agencies’ need for guidance, we address the following research question:


What are best practices of aid agencies for ensuring the success of outsourcing logistics to commercial LSPs in disaster relief?

Since academic literature in this regard is very sparse and did not address the same level of detail as our study (Bealt et al., 2016; Nurmala et al., 2017; Vega and Roussat, 2015), answering our research question required an exploratory approach. Moreover, the environment of disaster relief is well known for its lack of performance data and indicators (Abidi et al., 2014; Beamon and Balcik, 2008; Schulz and Heigh, 2009). Therefore, we decided to leverage the knowledge of humanitarian practitioners who have personal experience in engaging LSPs and, consequently, expert knowledge about advisable practices. As the responses of individuals are necessarily judgmental and situational, individual opinions need to be combined in order to develop a comprehensive picture. The Delphi method is particularly suited for exploratory research under such conditions (Okoli and Pawlowski, 2004; Akkermans et al., 2003; Meredith, 1993; Turoff, 1970). It is an anonymous and iterative technique for facilitating and structuring communication among a group of experts with the objective of either transforming individual opinions into group consensus or identifying systematic dissent among participants (Dalkey and Helmer, 1963; Hasson et al., 2000; Rowe and Wright, 2011). While common surveys are best at investigating “what is”, Delphi studies excel in asking “what […] should be” (Hsu and Sandford, 2007). Consequently, it is very appropriate for our type of research question and has already been leveraged in this context in other disciplines (e.g. Ager et al., 2010).

The Delphi method brings a number of further advantages for research in humanitarian logistics. First, it allows to account for the worldwide dispersion of humanitarian experts, since it can be conducted remotely and asynchronously. Second, the method does not require the expert panel to be statistically representative of any population, because the expertise of the panelists is more important than their number (Ludwig, 1997; Okoli and Pawlowski, 2004; Powell, 2003). This makes the approach more feasible in the humanitarian environment than large-scale surveys with representative samples, which are prone to struggle with the non-transparent group of existing aid agencies (Vega and Roussat, 2016). Third, the Delphi approach guarantees anonymity of responses (McKenna, 1994) and encourages humanitarian experts to express their true opinions. Given the competition between aid agencies and the importance of each organization’s reputation, practitioners might be reluctant to do so in case of non-anonymous research methods. Fourth, the Delphi method resolves group interaction effects such as the dominance of powerful individuals (Dalkey et al., 1972; Goodman, 1987; Rowe et al., 1991) and prevents the results from being biased toward humanitarian opinion-leaders, for instance, major aid agencies or UN organizations.

Despite the described benefits, only a few studies in humanitarian logistics have applied the Delphi method to date (Cottam et al., 2004; Mari Ainikki Anttila, 2014; Richardson et al., 2016; Sahebi et al., 2017). Therefore, as a secondary objective in addition to our content-oriented research question, we critically explore the applicability of the research method and its ontological, epistemological and methodological assumptions.

In summary, our research contribution is twofold. First, we determine a set of best practices, which can help humanitarian organizations to be successful when outsourcing logistics. To provide concrete guidance to humanitarian practitioners, our focus is on the compilation and discussion of detailed activities for implementing the best practices in disaster relief operations. Second, we shed light on the application of the Delphi method as a promising method for empirical research in humanitarian logistics. The critical debate of our research design is intended to support other researchers to successfully leverage the method for their own research. It has to be noted, however, that an all-embracing discussion of the Delphi method would require a designated paper. This paper focuses on the most important aspects in reference to our own study.

This paper is structured as follows. In the next section, we briefly review the relevant literature. In Section 3, we describe our research design and discuss its trustworthiness from a methodological perspective. In Section 4, we present the results of the empirical study, which we then discuss in Section 5 in view of existing literature. We close the paper with a summary and an outlook in Section 6.

2. Literature review

The existing literature on outsourcing in humanitarian logistics can be grouped into four categories. The first group of papers investigate the benefits, risks, barriers and enablers of cross-sector collaboration and partnerships between aid agencies and LSPs. Representatives of this stream of research are Balcik et al. (2010), Bealt et al. (2016), Cozzolino (2012), Maon et al. (2009) and Nurmala et al. (2017, 2018). The second group of studies focus on specific challenges of cross-sector partnerships, such as service contracts (Egan, 2010), cultural barriers (Dowty and Wallace, 2010) or cooperative purchasing initiatives (Pazirandeh and Herlin, 2014). The third group of articles explore the role of LSPs and outsourcing in humanitarian operations. Examples for this class of papers are Abidi et al. (2015), Baharmand et al. (2017), Cottam et al. (2004), Cozzolino et al. (2017), Heaslip (2013), Sánchez Gil and McNeil (2015) and Vega and Roussat (2015). The fourth group of studies are mathematical models which examine the efficiency of outsourcing (Nagurney et al., 2011; Wang et al., 2016), the tendering and selection process (Bagchi et al., 2011; Paul and Wang, 2015; Trestrail et al., 2009), and routing decisions of service providers (Huang et al., 2015).

While none of the previous studies has investigated how aid agencies can best ensure the success of outsourcing logistics, some of them mention relevant recommendations as a by-product of their research. Aid agencies should build close relationships with service providers to enable smooth planning and communication (Cottam et al., 2004). To achieve this, they should establish bi-directional data and information exchange, leveraging information technology and ensuring the protection of sensitive data (Balcik et al., 2010). In the best case, aid agencies should establish long-term strategic partnerships, which do not only involve the sharing of resources, but also the sharing of risks (Maon et al., 2009). Building this type of partnerships requires to understand which type of partnership mechanism applies for which situation (Nurmala et al., 2018) and to employ a high level of commitment, resolution, communication and partner focus (Cottam et al., 2004). Since different cultures and working styles can imply challenges for building partnerships (Nurmala et al., 2017), aid agencies should create internal awareness for cultural differences, for the expectations of the service provider and for the rationale behind both aspects (Dowty and Wallace, 2010). They should also adapt to the commercial sector by allowing more decentralized decision-making (Baharmand et al., 2017) and increasing the standardization of their processes (Bealt et al., 2016).

It can be beneficial for aid agencies to combine corporate resources (Maon et al., 2009) and engage in collaborative purchasing of logistics services (Pazirandeh and Herlin, 2014). However, mutual trust among the partners, a harmonized and coordinated tender process, an integrated approach to contracting, and harmonized agency processes and policies are important prerequisites for collaborative purchasing to be successful (Pazirandeh and Herlin, 2014). Alternatively, it can be advantageous to rely on 3PL or 4PL vendors for consolidating demand and managing the sub-contracting of providers (Cottam et al., 2004; Abidi et al., 2015). Aid agencies need to be clear and transparent on their requirements (Bealt et al., 2016) and consider a provider’s proven track record, existing relationship network, local knowledge and territorial coverage (Baharmand et al., 2017). They should not rely on single providers, but pursue multiple sourcing (Pazirandeh and Herlin, 2014) and build a network of partnerships (Nurmala et al., 2018). Finally, humanitarian organizations should ensure to have formal service contracts in place, which in the best case are set up in advance of a disaster (Baharmand et al., 2017), transfer the responsibility for the safety of goods to the provider (Baharmand et al., 2017) and are in line with local legislation and practices (Cottam et al., 2004).

In contrary to the field of humanitarian logistics, a number of dedicated studies have investigated success factors for outsourcing in the context of commercial logistics. They highlight the importance of planning the outsourcing engagement up-front and jointly with the provider (Goles and Chin, 2002; de Grahl, 2011; Hauptmann, 2007; Lambert et al., 1999; Selviaridis et al., 2008) and the benefits of establishing close working relationships with open communication and information exchange (Frankel et al., 1996; Goles and Chin, 2002; de Grahl, 2011; Lambert et al., 1999; Leahy et al., 1995; Qureshi et al., 2007; Selviaridis et al., 2008; Tate, 1996; van Laarhoven et al., 2000). They also emphasize the need for aligning expectations, clearly defining requirements and being thorough in the contract negotiations (Frankel et al., 1996; Hofenk et al., 2011; Lambert et al., 1999; van Laarhoven et al., 2000). Organizations should set up a written agreement or contract, which defines clear ground rules (Frankel et al., 1996; Tate, 1996) and includes provisions for adjustments and exit (Leahy et al., 1995; Selviaridis et al., 2008; Tate, 1996). Contracts and financial incentives are one possibility to manage provider opportunism (Hauptmann, 2007; Selviaridis and Norrman, 2015), especially when leveraged together with joint operating controls and a strong performance orientation (Lambert et al., 1999; Leahy et al., 1995; Qureshi et al., 2007; Selviaridis et al., 2008; van Laarhoven et al., 2000). Nevertheless, organizations should build relationships with mutual trust and understanding (Frankel et al., 1996; Goles and Chin, 2002; Lambert et al., 1999; Leahy et al., 1995; Qureshi et al., 2007; Selviaridis et al., 2008; Tate, 1996). This requires to select a service provider which is highly compatible with respect to capabilities, culture and reputation (Frankel et al., 1996; Goles and Chin, 2002; Hauptmann, 2007; Lambert et al., 1999; Leahy et al., 1995). Moreover, it requires the engagement to provide benefits to both parties and to include the mutual sharing of risks (Goles and Chin, 2002; de Grahl, 2011; Lambert et al., 1999; Leahy et al., 1995; Selviaridis et al., 2008).

In summary, the related literature on humanitarian logistics mentions a number of promising activities for aid agencies engaged in outsourcing, but provides a neither comprehensive nor integrated picture. The relevant literature on commercial logistics, in turn, gives a broader perspective, but describes only high-level factors, which are too abstract for being implemented by aid agencies. For example, the corresponding papers do not explain how relationships with mutual trust and understanding can be built or how a strong performance orientation can be established. Moreover, the insights from commercial logistics cannot be applied to the humanitarian context without adaptations because of the sector’s well-documented specifics (Kovács et al., 2009; Van Wassenhove, 2006). Accordingly, multiple researchers point out a need for dedicated research, especially empirical studies, on how aid agencies and LSPs can build successful outsourcing relationships (Bealt et al., 2016; Nurmala et al., 2017; Vega and Roussat, 2015). We address this gap with respect to both content and methodology by determining best practices for outsourcing logistics in disaster relief based on a Delphi approach and by compiling and discussing detailed activities for implementing these practices.

3. Research design

We conducted a two-round Delphi study between November 2017 and March 2018 with 31 experts from humanitarian organizations. It was administered electronically using the Calibrum Surveylet Software (Aengenheyster et al., 2017) and can, consequently, be called e-Delphi (Hasson and Keeney, 2011). After the finalization of the Delphi study, in April 2018, we conducted a complementary full-day focus group with ten experts from humanitarian organizations and two experts from LSPs (Morgan and Krueger, 1997; Morgan, 1996; Kitzinger, 1995). The purpose of the focus group was to perform a member check on the results, that is, to validate the outcome of the e-Delphi as well as our analysis and interpretation with subject matter experts (Creswell, 2009; Wallendorf and Belk, 1989; Lincoln and Guba, 1985).

3.1 The e-Delphi

The four basic principles of the Delphi method are an iterative and multistage process, some level of controlled feedback, the opportunity for individuals to revise their answers and a certain degree of anonymity (Linstone and Turoff, 1975; Hasson et al., 2000; Powell, 2003; Hsu and Sandford, 2007). While all Delphi studies share these common characteristics, the flexibility of the Delphi method has led to a high diversity of methodological variants (Hasson and Keeney, 2011). One important variant of Delphi design is the Policy Delphi, which differs considerably from the classical Delphi (Turoff, 1970). Its prime objective is not to seek consensus for decision-making or forecasting under uncertainty, but to identify and evaluate potential solutions for a selected issue and to reveal arguments for and against each solution (Turoff, 1970). Accordingly, its approach is rather qualitative, even if quantitative questions might be applied to evaluate the different alternatives. Determining best practices for outsourcing logistics in disaster relief requires to identify and evaluate different ways of managing activities such as provider selection, contract negotiation, performance evaluation or supplier management. We have, therefore, built our research design on the principles of the Policy Delphi.

Number of rounds

Delphi results tend to show only slight or even no improvements after the second iteration (Woudenberg, 1991). At the same time, the risk of biased results increases with the number of conducted iterations because of participant fatigue and drop-outs (Hasson et al., 2000). Accordingly, most Delphi studies consist of two or three rounds (Diamond et al., 2014). It is generally advisable to limit the Delphi study to the minimally sufficient extent and verify its results through follow-up explorations (Delbecq et al., 1975). We had already achieved a high level of consensus after the second round and perceived participant fatigue to become a relevant risk. Additionally, given our research objective, the explanations of panelists in reference to their second round judgments were considerably more important for our findings than minor changes of quantitative votes in a potential third round. Therefore, we limited the study to two rounds, knowing that any need for clarification could be addressed in the complementary focus group.

Expert panel

While there are no definite criteria for the definition and selection of experts (Hasson et al., 2000; McKenna, 1994; Hsu and Sandford, 2007), it is clear that the expertise and experience of the panelists determine the trustworthiness of the Delphi results (Hasson and Keeney, 2011; Hsu and Sandford, 2007; Lincoln and Guba, 1985). On a general level, experts are persons working in the relevant field with substantial knowledge of the topic under investigation (McKenna, 1994; Powell, 2003). For our study, we defined an expert as a humanitarian practitioner who had considerable and personal experience in outsourcing of logistics in disaster relief. Such persons possess access to privileged information which is difficult for other people to obtain and process. Accordingly, they are in a position to structure the field of action in a meaningful way and advise others on what to do (Bogner et al., 2014; Bogner and Menz, 2009; Meuser and Nagel, 1991, 2009). We requested potential participants to only register for the study if they possessed the described type and level of experience. In addition, we validated their expertise in Round 1, asking them to describe their personal experience with outsourcing along a number of dimensions. If their descriptions were coherently elaborated, their experience was seen as a sufficient basis for sharing information and providing advice to others (Stehr, 1994).

Most Delphi studies involve between 11 and 50 experts (Diamond et al., 2014; Hsu and Sandford, 2007). Since our research interest was to explore the perspective of aid agencies, we decided to not include any representatives of LSPs in the expert panel. For recruiting experts from aid agencies, we applied a mix of purposive sampling techniques: criterion-based sampling, snowball sampling and opportunity sampling (Patton, 2002). On the one hand, we approached all organizations which were listed as most important disaster relief agencies by Vega and Roussat (2016) based on the report Global Geneva’s 2015 Top 500 NGOs. Through either personal contacts or central mailboxes and phone lines we asked these organizations for employees who had personal experience with engaging LSPs in disaster relief operations and were willing to participate in our study. On the other hand, we identified relevant experts by asking our existing personal contacts at humanitarian organizations for further qualified participants within their network. In total, we recruited 31 experts from 24 humanitarian organizations who had on average 12 years of experience in humanitarian logistics, both at headquarters and field level (see Tables AI and AII). Their high level of experience, their qualifications and their long-lasting exposure to the problem under investigation contributed to the trustworthiness of our results (Hasson and Keeney, 2011; Lincoln and Guba, 1985).

Round 1

All recruited experts participated in the first round of the study, which was a qualitative online survey[1]. Before releasing the questionnaire to the panel, we pre-tested it with selected practitioners. The survey consisted of three parts (see Appendix 3): first, panelists were asked to describe a situation in which they had worked personally with LSPs, focusing on what they would do either similarly or differently under comparable conditions in the future. This framed the research subject and stimulated the link to the panelists’ experiences (Froschauer and Lueger, 2003). Second, they were requested to list success factors for outsourcing logistics in disaster relief in general. Third, they were prompted to provide specific recommendations with respect to the areas of provider selection, contract negotiation, performance evaluation and supplier management.

We analyzed the responses of the first survey, that is, the textual data, based on the principles of content analysis in NVivo. While NVivo, a leading tool for qualitative data analysis, was used to organize, store and retrieve the data, the actual analysis was carried out manually by the research team. Based on rules of procedure we assigned conceptual labels to the empirical data to explicate, summarize and structure the recommendations of participants (Mayring, 1991, 2000). The coding process combined both inductive and deductive approaches (Fereday and Muir-Cochrane, 2006). During deductive coding we assigned labels which were suggested by existing literature (theory-driven coding). When the suggestions of experts could not be referenced to existing literature, we applied inductive coding, that is, we assigned labels which were suggested by the data itself (data-driven coding). The coding process was organized as an iterative procedure in order to deal with inter-observer variability. First, three researchers with different professional backgrounds individually grouped the activities. Afterwards, multiple meetings were held to compare, discuss and consolidate the different classifications until there was perfect agreement among the researchers.

The analysis of the first round revealed a set of 113 activities which were recommended for ensuring the success of outsourcing by at least one expert. These activities were grouped into 12 higher-level categories, so-called practices, based on commonalities with respect to the objective of each activity.

Round 2

For validating the findings from the first round, we fed the anonymized results of our analysis back to the expert panel as part of a rather quantitative second online survey (see Appendix 4). This mix of qualitative and quantitative methods contributed to the methodological rigor of our study (Hasson and Keeney, 2011). As for the first round, we pre-tested the questionnaire before releasing it to the panel. In total, 27 experts participated in this survey (87 percent response rate). They rated both the importance of the 12 practices and the usefulness of the 113 corresponding activities on a four-point Likert scale. By limiting each page of the survey to a maximum of ten questions, we ensured that the response process was very manageable for participants. The design of both Likert scales was based on the recommendations by Turoff (1970). They featured an optimum number of alternatives, ensured consistent cognitive distances between adjacent pairs of points and prompted experts to take a clear position (Leung, 2011; Lozano et al., 2008). In addition to their quantitative rating, experts had the possibility to explain the reason of their judgment in form of an open comment (Singer and Couper, 2017).

Analyzing the replies of the second round, we calculated the mean rating of all items (activities and practices) and determined the level of consensus among the 27 panelists. We identified items lacking consensus by combining two well-established indicators (von der Gracht, 2012): interquartile range (IQR) and rwg. IQR measures the difference between the lower and upper quartiles. Following the recommendations from Raskin (1994) and Rayens and Hahn (2000), we marked all items with an IQR greater than 1 as “lacking consensus.” rwg is a measure of interrater agreement (James et al., 1993). In line with LeBreton and Senter (2008), we also labeled all items with rwg⩽0.30 as “lacking consensus”. In addition to the quantitative analysis, we applied content analysis to the participants’ qualitative explanations, following the same procedure as in Round 1.

Our analysis revealed a high level of consensus among experts; panelists had expressed controversial views for only 24 activities and 1 practice (see Section 4). There was a considerable level of doubt if a third round could resolve the remaining controversies (Woudenberg, 1991), especially because participant fatigue had reached a potentially risky level (Hasson et al., 2000). We, therefore, decided to terminate the e-Delphi after the second round and to use follow-up explorations for shedding more light on the remaining controversies (Delbecq et al., 1975). The complementary face-to-face focus group was used for this purpose.

3.2 The focus group

The focus group (Morgan and Krueger, 1997; Morgan, 1996; Kitzinger, 1995) was used as a member check (Creswell, 2009) and aimed to validate the outcome of the e-Delphi as well as our analysis and interpretation with subject matter experts. Member checks are a crucial technique for establishing credible and trustworthy findings (Lincoln and Guba, 1985). They are an opportunity to reveal latent assumptions of experts, challenge interpretations of researchers and obtain additional information from participants which may be stimulated by the playing back process. Accordingly, member checks should be treated as another source of data and insights (Silverman (2013) in reference to Fielding and Fielding (1986)). Used in this way, the focus group allowed us to gather additional textual data to enrich the findings from the e-Delphi. Additionally, it enabled us to understand in detail the reasons for the dissent among panelists regarding 24 activities and 1 practice. Debating both benefits and drawbacks of the particular items in a controversial, face-to-face discussion with selected experts allowed us to take a deep dive into the underlying assumptions and contextual relationships (Rowe and Wright, 2011; van de Linde and van der Duin, 2011).

In order to allow fruitful discussions within a diverse group of experts, we limited the number of focus group participants to ten and restricted participation to one representative per organization. Given the administrative and financial challenge of bringing together an internationally distributed group of experts for a face-to-face meeting, we recruited the focus group participants based on opportunity sampling (Patton, 2002). On the one hand, we invited all respondents of the second survey to the focus group. On the other hand, in order to account for the busy schedule of humanitarian practitioners, we offered panelists the opportunity to send equally qualified substitutes from their organizations in case they could not attend themselves. Focus group seats were awarded according to the sequence of registration. To ensure a representative pooling of judgments and experiences (Okoli and Pawlowski, 2004; Powell, 2003; Ludwig, 1997), we carefully monitored the process of participant registration, prepared to correct any potential bias in the sample through purposive sampling (Patton, 2002). In the end, a very competent and diverse group of experts was recruited for the focus group, consisting of five respondents of the second survey and five nominated substitutes (see Table AI). The new participants allowed us to reflect on the findings from an outside perspective and to incorporate new viewpoints into the results. In addition to the experts from aid agencies, we invited two representatives of commercial LSPs to the focus group in order to mirror the perspective of aid agencies. The presence of LSPs helped us to trigger truly controversial discussions. Through such discussions we were able to gain insights into, by then, tacit knowledge and assumptions of experts, which had not been accessible by the previous question-and-answer surveys (Rowe and Wright, 2011; van de Linde and van der Duin, 2011).

The full-day focus group consisted of multiple sessions. In each session, the research team first presented results from the two online surveys. In order to direct the participants’ attention to the content, methodological details were not explained. Afterwards, discussions took place among the experts. A member of the research team facilitated the discussions, summarized them and asked participants for validation or corrections. The focus group was audio-recorded, transcribed word-for-word and, afterwards, content-analyzed in NVivo, following the same procedures as for the textual data from the e-Delphi.

3.3 Methodological discussion

Even though the Delphi method entails considerable benefits, it is also subject to constant critique (Hasson and Keeney, 2011). Therefore, we deem it important to discuss the trustworthiness of our research design. Due to the qualitative and post-positivist nature of our study and its underpinning ontological and epistemological assumptions, the concepts of validity, reliability and objectivity are not applicable to our work (Day and Bobeva, 2005; Wallendorf and Belk, 1989). Instead, we will discuss the criteria of credibility, transferability, dependability and confirmability (Lincoln and Guba, 1985), which are well established for qualitative research. In doing so, we will focus on challenges which future Delphi studies in humanitarian logistics are likely to encounter and potential solutions to these challenges.


Discussing our results in the face-to-face focus group with a group of experts, partially internal and partially external to the Delphi, contributed to the credibility of our study in form of a member check (Wallendorf and Belk, 1989). This allowed us to take a deep dive into the panelists’ underpinning assumptions and logics and to refine the results accordingly (van de Linde and van der Duin, 2011). While this was an important part of our research, this might not be possible in other studies on humanitarian logistics due to reasons of time and cost. In the interest of credibility, we also applied triangulation across researchers and sources, for example, by performing content analysis within a heterogeneous team and by reviewing the results in the context of existing literature (Wallendorf and Belk, 1989). Still, incorporating further documents into the analysis, such as actual logistics contracts, could have been a promising extension of the triangulation approach.

Reflecting critically, it should also be noted that our study was executed in English. Since most of the participants were not native speakers, lingual misconceptions might have influenced the results. In general, misconceptions might also have occurred because of formulations in the questionnaires. For example, participants might have been misled by the perception of overlaps in the titles of the 12 practices. We tried to counteract by using simple expressions and unambiguous wording (Frewer et al., 2011; Parente and Anderson-Parente, 2011) and by performing pre-tests with selected practitioners for both surveys (Okoli and Pawlowski, 2004). The pre-tests also helped us to ensure that participants were able to comfortably handle the design, length and complexity of the questionnaires. While we perceived this approach beneficial, further measures could have been applied, for example, debriefing by peers during the development and analysis of the questionnaires (Lincoln and Guba, 1985).

Another limitation to credibility, which is inherent to all Delphi studies, is the influence of the selected consensus measures and the corresponding decision to terminate the study. A vast abundance of approaches exists and different measures might lead to different conclusions regarding the level of consensus (Diamond et al., 2014; von der Gracht, 2012). Our focus was not the achievement of consensus on a quantitative measure, but the identification of a set of important practices. Given this rather qualitative approach, the explanations of panelists in reference to their judgments were considerably more important for our findings than the choice of the consensus measure and the revision of the second round votes in a third round. For other applications in the field of humanitarian logistics, however, we recommend a critical consideration of this aspect. In conclusion, we have a good level of confidence in the credibility of our research. However, “as with most research, findings are never completely definitive but can be strengthened, clarified and enhanced by judicious and sound follow-up research” (Kennedy, 2004).


We established a heterogeneous expert panel by recruiting participants from diverse organizations and with a broad range of different experiences in disaster relief. Such triangulation through purposive sampling helped to enhance the transferability of results (Delbecq et al., 1975; Linstone and Turoff, 1975; Wallendorf and Belk, 1989). However, we faced limitations in this regard when trying to recruit experts from African or Asian countries, which were due to a lack of accessibility (for the Delphi) and high cost (for the focus group). This might have biased the results toward the perspective of European experts, even though our panel had extensive experience in the field. Future studies should put an extra eye on this topic and identify specific approaches for involving non-European panelists. As another measure for improving transferability, we sought for limiting exceptions throughout our study, which could define the boundaries of our results (Wallendorf and Belk, 1989). For this reason, we asked panelists to justify their judgments and provided them the opportunity to outline the reasoning behind their responses (Nowack et al., 2011). Panelists brought up some relevant boundaries for the transferability of our results on these occasions. For instance, the usefulness of selected activities could be influenced by contextual factors, such as the value of the service purchase, size and funding structure of an aid agency, the geographic context of the service (before or after point-of-entry) or the phase of disaster relief. However, such exceptions do not imply that our results need to be modified (Glaser and Strauss, 2017; Wallendorf and Belk, 1989). Even when exceptions represent a considerable number, there can be substantial evidence for the acceptance of the results (Lincoln and Guba, 1985). Reflecting all contextual factors of humanitarian logistics in the results would be nearly impossible. Instead, it is important to recognize that the application of the best practices by aid agencies might be subject to slight variations, depending on the contexts. Future Delphi studies could account for this challenge by working with multiple parallel panels of experts, which ensure homogeneity of each panel with respect to the important contextual factors.


Authors such as Cornick (2006) are confident that the use of a diverse and qualified expert panel would secure the stability and dependability of Delphi results. While we have designed our panel in exactly this way, we still consider it important to recognize the limitations of the Delphi method in this regard. According to the post-positivist paradigm, a single, objective reality does not exist (Lincoln and Guba, 1985). Both people and contexts change continually (Wallendorf and Belk, 1989). Accordingly, there is no guarantee that our results would repeat if we replicated the study with similar persons in a similar context. Therefore, Wallendorf and Belk (1989) recommend to explore the boundaries of dependability by applying a longitudinal approach and repeating investigations months or years later when things should have changed in various ways. By investigating these changes, researchers would be able to assess if the original theory is still applicable or needs to be modified. Given the dynamics of the humanitarian sector, different results in multiple years from now should actually not be surprising. Although this might appear a considerable limitation, this challenge is inherent to all qualitative research approaches. Furthermore, one needs to bear in mind that the Delphi method does not strive to generalize, but aims to provide a snapshot of expert opinion at a specific moment in time (Maceviciute and Wilson, 2009; Thompson, 2009). If researchers take this into account for their interpretation, and combine it with a longitudinal design as far as financial resources allow, dependability should not be a barrier for a wider use of the Delphi method in research on disaster relief.


Post-positivists philosophy also qualifies the requirement of confirmability: “There can be no absolute objectivity; at best the researcher can become conscious of and hopefully reduce […] biases” (Wallendorf and Belk, 1989). Therefore, a clear audit trail of data gathering and interpretation is one of the most important measures for enhancing confirmability in Delphi research (Rodgers and Cowles, 1993; Skulmoski et al., 2007). We established this by transcribing all interviews and focus group discussions, storing all survey responses in a database and recording notes on our analysis in NVivo. Also, to increase confirmability, we leveraged triangulation across researchers by involving multiple researchers in the planning and execution of interviews and focus groups as well as the analysis of textual data (Wallendorf and Belk, 1989). While we consider these techniques appropriate, future studies based on the Delphi method could, nevertheless, apply further measures in this regard. For example, they could make use of external auditors which are supplied with all raw materials as well as their interpretation (Wallendorf and Belk, 1989). Moreover, they could formally determine inter-coder reliability in order to assess the confirmability of the coding process (Mayring, 2000).

4. Results

In the first survey, 31 experts recommended 113 activities for ensuring the success of outsourcing logistics. As described in detail in Section 3, these were grouped into 12 best practices and presented for further evaluation in the second round. In that survey, 27 panelists rated the importance of each practice on a four-point Likert scale (not important, somewhat important, important and very important). Table I shows the mean rating of each practice. Three practices were rated very important (mean rating⩾3.50) and nine practices important (2.50⩽mean rating<3.50). In general, there was consent among panelists (IQR⩽1 and rwg>0.30). Only for the importance of practice P07, no consent was achieved.

In total, 27 panelists also rated the usefulness of each activity on a four-point Likert scale (not useful, somewhat useful, useful and very useful). Table AIII shows the mean rating of each activity. While 35 activities were considered very useful (mean rating ⩾3.50) and 71 useful (2.50⩽mean rating<3.50), 6 activities were found to be only somewhat useful (1.50⩽mean rating<2.50) and 1 was rated not useful at all (mean rating<1.50). For the majority of activities (89 activities), experts agreed on their usefulness (IQR⩽1 and rwg>0.30). For 24 activities, some level of controversy existed among panelists. In some cases, experts did not agree if an activity was useful or very useful. In other cases, experts had opposing views. While one group considered an activity very useful, another group deemed it not useful at all. These items were discussed in detail during the focus group in order to identify the reasons for this controversy.

We will now describe in detail the 113 activities (Table AIII) which were suggested by the panel for implementing the best practices (Table I). We will use expert quotes from the e-Delphi and the focus group to exemplify the recommendations. Where applicable, we will outline the reasons for existing controversies.

4.1 Rank 1: ensure both compliance and efficiency in the procurement process (P03)

The most important practice, according to the experts, is to ensure both compliance and efficiency in the procurement process and to properly balance both. Aid agencies face two requirements, which are often conflicting: speed and compliance. On the one hand, their “main objective is to save lives” (E19)[2] and any delays or inefficiencies in the procurement process put this target at risk. On the other hand, many humanitarian organizations “are donor driven agencies [and] have to, even in the emergency, maintain a level of compliance” (E25) with donor requirements, for example, a fair and transparent, but also time-consuming tender process.

Many times, “the good selection, which […] fulfil[s] all the necessary accountab[ibility] procedures, is in direct opposition to the tight and quick deliveries” (E35). Therefore, experts recommend to align procurement procedures to donor requirements and public procurement regulations, but to allow derogations from these procedures in the case of “sudden-onset scenarios or where [a] beneficiary’s life is at risk” (E21). For example, long-term agreements should be tendered openly and publicly and be awarded based on the decision of a multi-person tender committee. Short-term contracts in the context of emergencies, however, can be awarded either based on restricted tenders, where a request for proposal is sent to a limited number of suppliers only, or even without competitive bidding based on direct negotiations with selected providers. This is “is quite often by many donors accepted [sic.]” (E35) since they “understand the concept of [a] life-saving emergency” (E25). The challenging part is “to determine when the emergency stops, and at which point the organization has to be ready to roll-out the proper [procedures]” (E25). The danger “is that if you start […] an operation based on derogations, there is a tendency that the organization continues to use derogations, and derogations, and derogations, and never takes that next step to operate in the correct manner” (E5). For example, “in the Syria response […] [t]here were agencies that were keeping emergency response for two years” (E25). Derogation rules can be abused by people in situations of self-imposed urgency, which are due to internal shortcomings and not the disaster itself. Therefore, experts recommend to strictly regulate the use of derogation rules by defining appropriate approval workflows, for example, “senior managers to look into the urgency of the request to determine whether there is a justification” (E25). Moreover, top management should clearly define the duration of the emergency status, potentially also based on external reports. One agency, for instance, only allows exceptions due to emergencies if the particular location at the given time is formally recognized by the European Commission as a crisis situation. Besides defining clear exception rules, organizations should carefully document every purchase from request to decision, also in emergencies. Finally, experts emphasize that, given appropriate preparation and planning (see Sections 4.4 and 4.5), compliant procurement is also possible under time pressure: “I think in 99% of our businesses you can tender this stuff” (E6).

4.2 Rank 2: use a detailed written contract as safeguard (P06)

Contracts serve as safeguards for aid agencies by documenting the agreement with the LSP and providing the legal ground for enforcing the agencies’ claims in case of any opportunistic behavior of the provider. Furthermore, organizations can use contracts as a manual of cooperation and to establish “a very clear understanding and description of the mutual expectations” (E23). Accordingly, experts deem it very important to set up a detailed and written contract: “Contracts […] with logistics service providers, especially in the beginning of […] operations, must be very carefully written” (E2), because “what is not written, is not in the contract” (E19).

According to the experts, the contract should explain the context of the operation and the specifics of a humanitarian mission, such as urgency or uncertainty. It should specify the exact scope and duration of the service, clarifying “the differentiation between included and not included” (E27) activities to forestall any discussions about gray areas of responsibility. In addition, agencies should outline the service provider’s supplementary responsibilities, such as documentation or reporting, and describe guidelines for the collaboration, such as planning or communication procedures. Attaching a code-of-conduct, data protection rules or non-disclosure agreements are considered further useful measures in this regard.

The contract should also include performance indicators and service-level agreements. Only in this way, agencies can “assess […] providers’ service level and performance against contractual agreements” (E24) (see Section 4.8). These should be complemented by specifying the consequences of a failure to fulfill the service-level agreements (e.g. penalties or liquidated damages). Likewise, provisions for other adverse events, such as product damages or theft, should be recorded in the contract. Regulating the responsibility of securing insurance coverage is paramount in this regard.

Further important elements of the contract are the structure of service fees and the applicable payment terms. In this context, aid agencies should, for instance, set safeguards against price increases due to unforeseen circumstances: “Price adjustments after finalizing the contract must not happen. The provider has to handle changing conditions” (E18). While this approach should be balanced with the objective of creating win–win situations (see Section 4.11), conflicts are occasionally inevitable. Therefore, sufficient provisions for adjusting or terminating the contract are advisable. Likewise, the contract should outline the approach to dispute arbitration, the governing law and the court of jurisdiction.

4.3 Rank 3: be detailed in the request for proposal or request for quotation (P04)

Panelists emphasize that a clear and precise service request is the foundation for a successful outsourcing engagement: “If you do a proper analysis of your needs, and you specify very well what […] you expect from the contractor […], that’s […] going to lead to [a] clear contract and contractual obligations” (E25). Based on a clear understanding of their own requirements (see Section 4.5), agencies should explain these in detail to the provider: “the more info [is] shared, [the] less issues come up at the negotiation and signing stage” (E24). In this regard, it can also help to attach a draft contract with standard terms and conditions to the service request since this “avoids any surprises down the road” (E5). This is, in particular, important for services at the field level; international service providers often prefer to use their own standardized contracts. Experts highlight, furthermore, that is not always “possible to give all exact details upfront” (E21) as the requester might not have all required information either. In these cases, it is important “to give firm and good indications, but [to allow for] some flexibility” (E21).

In addition to detailed requirements, experts recommend to highlight the context of the operation and the specifics of a humanitarian mission in the request to the provider. Moreover, in order to establish transparency, agencies should clearly explain the bidding and selection process (e.g. administrative requirements, timeline and selection criteria). Attaching standardized bid templates and questionnaires for additional information (e.g. customer references) enables a smooth and efficient process for both agencies and service providers. The latter is very important, because “many vendors find the detailed documentation and need for information from them an obstacle” (E28). Extensive bidding requirements can lead to lower response rates from LSPs and hinder the selection of the right provider (see Section 4.6).

4.4 Rank 4: prepare for the engagement of logistics service providers (P01)

Panelists consider it important to make sufficient preparations for the engagement of LSPs: “Bringing things into the country […] has to be arranged before the emergency actually occurs” (E25). Preparation can take different forms. Organizations should, first of all, develop guidelines, process documentations and standard operating procedures, which define how the organization handles outsourcing, for example, regarding the procurement of services or the setup of contracts. Especially in emergencies, such documents “help to avoid confusion” (E22) and allow “to be efficient” (E34), because “if there is no documentation, […] it is going to be a big chaos” (E17).

It is even more important, though, that these procedures are actually put into practice and “not merely for documentation” (E31). This is one of the reasons why experts advise organizations to sufficiently train their employees, for instance, with respect to tender processes or vendor management. Establishing a group of experts in central procurement offices can help in this regard. However, it is highly recommended to also sufficiently train field staff which is interacting with LSPs. As Expert 28 explained, “signing contracts for those people who’re not good professionally is very, very difficult because they don’t understand it. […] [E]ven the tender processes are sometimes too complex for them.” Regular team meetings for discussing “lessons learnt” are an important addition to formal training sessions.

Personnel development should be complemented by developing appropriate tools and templates, for example, standardized contracts, checklists for performance evaluations or provider databases. The latter should be fed by regular assessments of service provider markets, in which new potential providers are identified, market prices are compared and cost structures of providers are analyzed. Expert 8 underlined their importance: “We reduced our cost […] by one million US Dollar over the first year after we started actually doing market assessments.” At best, organizations do not only assess service provider markets, but also run pre-qualification procedures and set up framework agreements covering “as many eventualities, situations [and] supply chain requirements [as possible]” (E21). These long-term contracts usually do not contain any volume commitments and sometimes not even fixed prices. Still, “longer-term framework contracts [are] a very useful tool” (E35). Superseding repetitive and deferring procurement procedures, they allow to “activate the services with short notice” (E21) and with “less administrative [effort]” (E35). Finally, experts recommend exchanging best practices and templates with other humanitarian organizations and engaging professional external support, for example, experts for the tender process or contract design.

4.5 Rank 5: set up and maintain a plan for the engagement of logistics service providers (P02)

Outsourcing does not only require appropriate preparation (see Section 4.4), but also sufficient and adequate planning. It starts with aligning the outsourcing approach to the overall logistics strategy and developing an exact understanding of service needs and requirements. Agencies should “identify gaps in the infrastructure” (E26) and determine how LSPs can best help to bridge them. Conditions in disaster relief operations often change considerably within short time. As one consequence, agencies should budget for the handling of challenges during outsourcing engagements, as those are almost inevitable in the light of rapidly changing circumstances. As another consequence, organizations need to prepare for these eventualities through scenario planning. “If you have the plan in the drawer and [even if it] is 6 months old […], at least you have something, you have a start” (E5). This also implies to source multiple alternative providers for the same service in order to have sufficient fallback options when single providers are not able or willing to provide a service.

Experts recommend complementing these activities by regular risk assessments, which analyze the fit between the outsourcing structure and the potential development of the operational environment. Aid agencies should also involve contracted providers into this process in order to identify “any constraints from their side” (E25). For example, they should “ask LSPs to share their contingency plans for […] any unwanted situation” (E22), such as alternative routes of transportation. Experts even recommend agencies to test these backup plans together with providers, for instance, by routing single shipments on different paths, whenever time allows: “If it’s working, we know that we have a plan B. If it’s not working, […] we have to search for something else” (E10). In addition, organizations should agree with existing partners upon prices for potential future service requirements, as long as the principles of fair and transparent procurement are not violated.

Finally, planning enables aid agencies to negotiate with providers based on consolidated service demands, for example, aggregated across regions, and to coordinate service purchases with other aid agencies. Coordination, at worst, reduces competition for limited capacities. At best, it offers the opportunity to tender services jointly with other agencies increasing an organization’s bargaining power vis-à-vis service providers.

4.6 Rank 6: take actions to ensure the selection of the right provider for the right task (P05)

Experts consider it important to invest time and efforts in the selection of the right service provider, since “without the right provider, [the] right service cannot be ensured” (E19). The definition of “the right provider” is, however, very context-dependent, since no provider can perform best in all situations. Aid agencies should assess, which type of service provider is best suited under which circumstances, and, instead of just selecting one provider, source a “suit of providers which you can call upon” (E13). This allows the flexibility of working with the best fitting provider in each situation (see Section 4.5).

Experts provided specific recommendations for the selection of the right provider for the right task. Organizations should approach as many LSPs as possible with a service request and allow providers sufficient time to respond. In the case of open tenders, good media support should be secured, for example, through publications in well-established newspapers. When choosing between different service providers, information becomes paramount. However, this can be a major issue in disaster relief operations: “The main challenge is, actually, that we don’t have information to make the right selection” (E8). Accordingly, experts recommend various activities to fill this void of information. Particularly less known providers should be asked to provide customer references together with their proposals. These documents offer the possibility to obtain information about the provider directly from other organizations. In general, aid agencies should make use of their network. By reaching out to other aid agencies, local authorities, donors or product suppliers, they can benefit from a “solid base of […] sources that are already verified by somebody else” (E25) and avoid negative surprises. Experts emphasize the importance of the Logistics Cluster in this regard, because “one of [its] key mandates is information management” (E8).

Once provider proposals have been submitted, requesting bid presentations and questioning LSP representatives in person are considered useful by experts. These activities help to identify the true logistics capability of the provider. Measures such as site visits and physical checks of the provider’s assets have the same objective. Without activities of this sort, agencies face the risk of providers not being able to perform the service at the promised service level and price. Experts explain that “[w]ritten proposals contain false information just to qualify [for] the minimum requirements” (E22) and that providers “on purpose underrate their offers to beat their competition” (E25) or because they “do not know their own costs” (E7). The capability of providers can also be screened by first testing the contractors under less critical conditions before involving them in important operations. Another very useful activity is, according to experts, the execution of background checks on the provider, for example, regarding its independence, legal compliance or financial health. This is particularly critical at the field level, where providers with dubious practices or links to military groups can not only harm an agency’s reputation, but also endanger its employees.

4.7 Rank 7: avoid unnecessary liabilities and risks (P08)

The conflicting goals of service providers and the unpredictability of disaster relief operations inevitably lead to risks for aid agencies engaged in outsourcing. Therefore, panelists regard it as important to avoid unnecessary liabilities and risks. While longer-term contracts help to build relationships with providers (see Section 4.9), experts consider them contraindicated in a number of situations. For example, in the beginning of a relief operation, “when suppliers […] raise the prices and it’s expected that a few months after[wards] the market will start settling down” (E25). Also, in “volatile economic situations, especially [with] rapidly fluctuating exchange rate[s]” (E31) or when “working for the first time with a service provider” (E15), contracts should be set up as short as possible. However, even in these cases, organizations should “make sure that it is more than just a one-time commercial contract, so that the service provider can be used […] in the future” (E2).

If longer-term contracts are set up, organizations should ensure a sufficient level of flexibility by avoiding any kind of volume commitments or exclusivity clauses. Terms of the latter type might require agencies to treat a LSP as “sole source provider unless there is a capacity issue” (E28). Such agreements are considered risky, because “capacity is different to capability” (E28) and “complementarity of coverage by multiple providers” (E38) is highly important (see Section 4.5). Panelists, furthermore, recommend specifying sufficient provisions for contract termination, for example, due to funding issues or the end of a mission, and to agree upon future price adjustments according to prevailing market prices.

In general, oral communication should always be followed-up in written to have proof of the conversation. This is especially important in situations of high urgency, when organizations might be forced to “hire someone […] without […] a formal contract” (E16). In such cases, experts recommend to, at least, request a written commitment from the provider to adhere to the agreed service fees. If a formal contract is set up, it should either be reviewed by a legal expert or be based on a standard template released by the legal department.

Finally, in particular at the field level, it is important to avoid communicating sensitive data to providers, for example, beneficiary lists or budget details, as this could cause severe security issues. Additionally, agencies should avoid too close personal relationships with providers and always “remain [in] professional distance” (E8). In some cultural areas, “for a private company […] it’s normal to give incentives” (E17) and organizations should make sure to avoid any impression of favoritism.

4.8 Rank 8: run a formal process for performance evaluation (P09)

Bad performance of service providers can be a serious problem for aid agencies, affecting its reputation and the well-being of beneficiaries. “The quality of services may vary not only [within] the country context, but also [for] the same supplier over time” (E16). Therefore, panelists deem it important to run a formal process for evaluating the provider’s performance. According to the experts, there are two main dimensions of performance evaluation: continuous quality control and regular performance reviews. Especially “when working with local 3PL firms, quality control is a major issue” (E6). Agencies should closely monitor and track the execution of activities by the service provider, always involving those employees who have the best visibility of the service. According to Expert 10, this is “something that we are daily doing by calling our colleagues.” Likewise, inspections and audits on service quality and compliance are advisable, although they should be applied carefully as they can “often cause unnecessary delays” (E22).

Besides performing continuous quality control, organizations should conduct a formal review of performance, taking place at specific milestones or in fixed intervals. On these occasions, agencies should use key performance indicators (KPIs) to evaluate the provider’s performance against benchmarks and contractual agreements. However, organizations “need to be careful with KPIs, [and] the key is to use only a few” (E28). Furthermore, they need to account for the fact that disaster relief “is not normal, it’s generally chaos; […] your normal business-as-usual KPIs just don’t apply and shouldn’t apply” (E21). Accordingly, indicators which measure the humanitarian performance of the provider are required. They should cover both quantitative factors, such as timeliness or cost, and qualitative aspects, such as transparency or agility. It can also be helpful to involve the service provider into the definition of the performance metrics as this facilitates an “alignment of expectations between the parties” (E21). Aid agencies should, moreover, maintain a record of performance for each provider, which is fed into a service provider database and stored across operations and engagements (see Section 4.4). This allows to use the information not only for the single performance review, but also “as basis of continuous improvements” or as a “negotiation tool” (E16) for future negotiations. Finally, despite the need for performance management, experts emphasize the importance of “keep[ing] the focus on impact of programming rather than donor priorities [or] shipping metrics” (E29). In the end, the “main performance indicator [is] successful delivery of relief items to the beneficiaries for life saving operations” (E2).

4.9 Rank 9: build mutual understanding and trust with provider (P10)

According to the experts, high levels of trust and understanding between an aid agency and its service providers bring significant advantages. For example, providers are more likely to go beyond their contractual obligations in critical situations when they perceive the engagement to be a trustful partnership: “the relationship is everything” (E28). Consequently, panelists consider it important to build mutual understanding and trust with providers.

Building trust starts during the tender process and “the best tendering approach should involve establishing a longer-term partnership” (E7). Therefore, organizations should avoid too much variation in their supplier base: “We only have about 4 service providers […] that we work with on a consistent basis and have long standing relationship[s] with all of them. […] [T]his aspect of close relationships helps us work together towards a common, mission-oriented goal” (E29). However, this approach “really clashes against this fair, open, transparent scheme” (E5) which is requested by donors and limits the options for building relationships: “Even though we had a fruitful relationship [with a provider], we still have to treat them like everybody else” (E5). Helpful and still compliant measures are personal invitations to public tenders for preferred partners, in order to ensure their participation, and feedback to loosing tenderers, in order to increase transparency and build rapport.

When working for the first time with a provider, aid agencies should explain the specifics of the humanitarian context to the provider’s staff and define relationship managers on both sides, because “building trust in the beginning […] often depends on individuals in [sic.] both the company side and the NGO side” (E14). In this regard, it is also advisable to have selected staff of the provider working within the aid agency for some time, or vice versa. This does not only “bring […] special expertise into the NGO” (E3), but also helps to gain “a better understanding [of] the internal dynamic procedures of the partner” (E35). Due to the risks of close personal relationships (see Section 4.7), panelists deem this measure more suitable at the headquarter level. Experts, furthermore, recommend setting up contracts with longer durations as long as the circumstances allow it and sufficient provisions for termination and adjustments are included (see Section 4.2). In particular for local services, they should review the contract jointly with the LSP, because “in many cases […] suppliers do not even read the contract, they just sign any document to get the contract” (E19).

Overall, experts emphasize that building trust is a “two-way street” (E25). Therefore, agencies should always keep their own commitments, for example, with respect to payments. Moreover, they should be flexible if changes to agreed terms are required. They should “understand the situation of the service provider” and “have [a] fair manner of treating the partnership, […] not necessarily sticking 100% to the contract” (E8). Aid agencies should, additionally, share projections on future service demands with selected providers to signal their interest in long-term relationships.

4.10 Rank 10: work with provider as an integrated team (P12)

“When you can engage the service provider as a core team member, not as a general support entity, [it] offers the most” (E22). Consequently, panelists consider it important to work with providers as an integrated team. Aid agencies should, first of all, define contact persons and corresponding backups, which are available for phone calls during 24 hours a day. Moreover, they should target to work face to face with the service provider whenever possible, because “as soon as you are working face-to-face for some time, you build trust, you build understanding, and you have [a] better relationship” (E14). However, at the same time, they should take measures to avoid any impression of too close personal relationships or favoritism, especially at the field level (see Section 4.7).

Once the tender process has been finalized, experts recommend involving the provider in the logistics planning of an operation (see Section 4.5). Furthermore, aid agencies should proactively share all relevant information with the provider and, in general, communicate frequently, honestly and transparently. In time-critical operations such as disaster relief “the common level of agreement is vital” (E22) and “regular communication […] may minimize the ambiguity and speed up the process” (E31). Additionally, organizations should cooperate with the provider to find joint solutions for challenges and even “help it to best being able to fulfil its contract in a safe and practical manner” (E27). Treating the provider as a team member, according to panelists, also requires investing in the development of its capabilities. Therefore, organizations should not only provide continuous performance feedback to providers and regularly discuss “lessons learnt”, but also involve the vendor’s staff in internal trainings on standard operating procedures and best practices: “for example, we train the drivers in how to drive economically and ecologically” (E2).

Finally, information technology should be leveraged to enable smooth cooperation. Data should be exchanged electronically between IT systems, in best case through electronic data interchange (EDI). If regular information exchange is required, for example, in the case of warehousing services, experts consider it useful to give the provider access to the aid agency’s IT system, as long as the system allows to restrict the provider’s access to the required modules and functionalities.

4.11 Rank 11: design engagement as win–win situation (P11)

In general, panelists agree that the engagement should provide sufficient benefits to both sides, because “partnering with a LSP offers much more [when both] NGO and LSP […] benefit” (E28). If “both the client and the provider are satisfied with the agreement, [they] will do all efforts to ensure its success” (E23). Accordingly, participants deem it important to design the engagement as win–win situation. However, a lot of controversy exists with respect to the best approach for achieving it. A key reason might be the “historic relationship the humanitarian world was having with the private sector, which has been not too trustful and [has] been very much looking at the private sector [as] someone who just wanted the profit and just wanted to cheat” (E8).

There are still sceptical opinions: “commercial profit is by all means enormous” (E19) and “when emergency situation[s] occur […] genuine competition seem[s] to disappear” (E21). Nevertheless, panelists consider it useful to agree upon fair service fees which allow for adequate provider profits, because “we engage with the commercial world which needs to generate profit” (E21). Likewise, experts recommend both the use of cost-plus contracts, which link service fees to the actual costs of providers, and the specification of price adjustments according to official cost indices. These are not only fair approaches, which safeguard providers against increasing costs, but also avoid that LSPs charge premium fees to hedge risks or that a provider “reduces the quality of the service […], because he has to get that [increasing] cost from somewhere” (E16). While experts agree on the advantages of index-based price adjustments, they highlight that the approach is “not easy to manage” as it’s “not very popular when the fuel price goes down and you have to [lower] the rates” (E8).

Panelists also recommend acknowledging a good performance of providers by providing a letter of reference, which they can use in future tenders. Some experts suggest making faster payments, directly extending contracts in case of satisfactory provider performance and guaranteeing fixed minimum payments to providers, independent of the transaction volume. These activities are, however, not considered useful by other panelists. Similarly, a lot of controversy exists with respect to performance-based service fees, which include a bonus for performances above target. Some think that “offering more money for better service never works” (E22) and that it would be very difficult to justify to donors “that we have paid somebody more than what they asked” (E25). Others argue that “imposing penalties for suppliers is default, but giving incentives is not” (E19) and actually “carrot and stick both should be [t]here” (E19). Also, letting the provider benefit from the humanitarian image through publicity is seen controversially. While it can help to get better and cheaper services, it might not be possible due to donor requirements or security issues. Some argue that, if at all, it should only be granted in case of free-of-charge services.

4.12 Rank 12: set up financial incentives to align goals of service provider (P07)

The importance of setting up financial incentives is seen controversially by experts. While some consider it not important or only somewhat important, others deem it important or even very important. This difference in judgments appears to be due to the complexity and context-specificity of defining appropriate financial incentives.

Without controversy, experts recommend defining adequate documentation by service providers as precondition for payments. Failure in this regard, for example, missing waybills, can lead to serious conflicts with donors. Organizations should also avoid payments in advance of the service, wherever possible. However, such payments may be required, because “in certain cultures, in certain places […] you have to deposit the amount” (E5) and an “advance payment is the only way to get a service” (E8). Likewise, such payments might be necessary if the provider is “just a small company, with little cash flow […] [and] cannot start without this payment” (E35). Therefore, aid agencies should use their best judgment and accept that this is a “risk you have to take […] [and] explain […] to the donors” (E5).

There are also opposing views regarding the use of open-book contracts, in which a provider reveals its actual costs to the aid agency. Some experts have experienced them to work extremely well, because they offer a high transparency and reduce the risk of being cheated. Other experts emphasize, though, that “depending on the culture […], [open-book contracts] may be synonymous to no trust” (E16) and can negatively affect the partnership (see Section 4.9). Furthermore, facilitation payments or bribes can cause challenges for open-book contracts: “if you need to get things through certain areas, customs, there will be certain fees […] that we really can’t substantiate […] because it’s an ineligible cost [for donors]” (E5).

As another financial incentive, experts suggest guaranteeing contract extensions in case of satisfactory provider performance, even though an agreement on the definition of “satisfactory” can be difficult to achieve. Finally, panelists recommend performance-based service fees, i.e. fees that are reduced if performance is below a pre-agreed target. Such penalties can, for example, be defined for late deliveries. However, “you can only apply [penalties] to factors within the 3PL control, which may be limited in a humanitarian context” (E7). In general, penalties conflict with the objective of building mutual trust and understanding (see Section 4.9). Therefore, penalty terms “need to be fair and realistic” (E21) and organizations should carefully weigh their application: “penalties are often […] part of the contract, but […] rarely applied […]. Best is to be able to work with the supplier” (E24).

5. Discussion

In this section, we reflect on the results in view of relevant literature on commercial and humanitarian logistics, which has been presented in Section 2. According to the expert panel, the most critical practices for ensuring the success of outsourcing logistics are balancing compliance and efficiency in the procurement process (P03), using detailed written contracts (P06), and issuing detailed requests for proposal or quotation (P04). All of them are considered very important (mean rating⩾3.50).

Practice P03 is not only considered the most important one, but also differs in another regard from the remaining practices: it was the only practice which we could not map during the process of deductive coding, at least to some extent, to success factors from commercial logistics literature (see Table II). The practice seems to be irrelevant in the commercial context (Selviaridis and Spring, 2007). While commercial enterprises are independent of donations, donors play a significant role in the humanitarian environment (Wakolbinger and Toyasaki, 2018). Their requirements have to be considered in many ways when outsourcing logistics, for example, with respect to the applicable procurement procedures, remuneration forms or payment terms. They impose “a major constraint on our ability and as organizations” (E5). If their requirements are violated, donors can withdraw their funding causing serious financial problems for agencies. As, simultaneously, time pressure and speed play an outstanding role in emergency operations (Van Wassenhove, 2006), it appears plausible that experts consider Practice P03 a very important addition to the elements from the commercial sector, and, in fact, the most important of the 12 best practices. Quite contrary to the experts’ view, the importance of compliance requirements for outsourcing has not been stressed a lot by literature on humanitarian logistics. In general, academic literature has not investigated sufficiently the implications of donors on operational decisions (Burkart et al., 2016). Our results, similarly to Bealt et al. (2016), suggest future research in this regard.

Different than for Practice P03, the relevance of the two other very important practices, P06 and P04, is also highlighted by the existing literature. It is recognized that commercial enterprises should align expectations, provide well-defined requirements and be thorough in the negotiation of a formal contract, which defines clear ground rules and includes provisions for adjustments and exit (Hofenk et al., 2011; Selviaridis et al., 2008; van Laarhoven et al., 2000; Lambert et al., 1999; Frankel et al., 1996; Tate, 1996; Leahy et al., 1995). Also humanitarian organizations should set-up formal contracts, ideally in advance of disasters, and be clear in their requirements toward service providers (Baharmand et al., 2017; Cottam et al., 2004; Bealt et al., 2016). As humanitarian organizations and LSPs have inherently diverging objectives and motivations (Nurmala et al., 2017), aligning expectations is potentially even more important than in the commercial context. A detailed service request and a detailed contract, as a manual of cooperation (Hauptmann, 2007), can help to clarify both requirements and expectations. Our results go beyond these high-level insights and provide clarity on how aid agencies can reflect these objectives within the content of contracts and the process of service requests. Specific requirements of the humanitarian context become obvious on this level of detail. For example, at the time of the service request, not all required information might be available, and both the aid agency and the provider need to show flexibility in negotiating the service.

A major recurring theme in literature on humanitarian logistics is the importance of preparedness for disaster relief (Jahre et al., 2016). This also applies for the field of outsourcing. By investing in training, relationship building or the setup of service agreements in advance of relief operations, the engagement of the private sector in the response phase can be improved considerably (Bealt et al., 2016; Quero, 2012; Tomasini and Van Wassenhove, 2009; Zyck and Kent, 2014). Our results mirror this perception: according to the expert panel, adequate preparation (P01) and planning (P02) are important practices for ensuring the success of outsourcing. Experts suggest that preparation should cover guidelines, documentation, training, staff development, tools, templates, information gathering and relationship building. While aid agencies recognize the importance of these activities, most of the organizations never or only occasionally make such preparations (Bealt et al., 2016). Donor priorities and a corresponding lack of preparedness funding are certainly among the main reasons for this apparent contradiction (Jahre et al., 2016). Therefore, low-cost preparedness activities with a high return on investment are of special interest to aid agencies. Our results contain some promising recommendations in this regard, for example, regular assessments of provider markets, which require moderate effort but can lead to significant cost reductions.

The literature, both from the commercial and humanitarian field, also highlights the importance of organizations building close relationships with LSPs and establishing long-term partnerships (Cottam et al., 2004; Maon et al., 2009; Bealt et al., 2016; Nurmala et al., 2017). Especially trust is frequently mentioned as one of the most important preconditions for successful outsourcing relationships (Bealt et al., 2016; Frankel et al., 1996). Our results, on the one hand, confirm this judgment by literature. Experts consider it important (2.50⩽mean rating<3.50) to build mutual understanding and trust with providers (P10), to work with providers as an integrated team (P12), and to design outsourcing engagements as win–win situations (P11). On the other hand, our results relativize the role of partnerships and trust to a certain extent. Panelists perceive these practices (P10, P11 and P12) considerably less important than, for example, setting up detailed and formal contracts.

The relationship of trust and contracts is certainly a controversial one. It is subject to a continuous academic debate since Macneil (1978) first differentiated classical, neo-classical and relational contracts. It remains often contested if contracts and trust function as complements or substitutes, and which of the two is more important (Boyson et al., 1999; Frankel et al., 1996; Hofenk et al., 2011; Poppo and Zenger, 2002; Spekman and Davis, 2004). Trust is a very complex concept itself, subsuming deterrence-based trust, calculus-based trust and relational trust (Rousseau et al., 1998). Our results suggest that in the humanitarian context contracts are more important, but still need to be complemented by trust: “One thing is to have a good contract, which is the basis and you need to have that, but we need to live beyond the contract in a professional manner” (E5).

There are various potential reasons for why experts consider detailed contracts more promising for ensuring the success of outsourcing logistics than establishing trust-based partnerships. The main one appears to be the difficulty of building trust in disaster relief. Hastily formed networks and the lack as well as high turnover of adequately trained staff cause big challenges for developing relationships (Kovács et al., 2009, 2012; Pettit and Beresford, 2009; Tatham and Kovács, 2010). This is specifically true for relationships between aid agencies and commercial actors, which are even more challenging because of differences in culture and working styles (Nurmala et al., 2017). One exemplary consequence is humanitarian practitioners fearing that commercial LSPs might abuse the context of disaster relief for making excessive profits (see Section 4.11) while the financial resources of aid agencies are scarce (Kovács et al., 2009). It appears that aid agencies first need to address the foundations for building trust before they can more comfortably substitute formal contracts by relational ones. For example, they would have to increase their level of communication with partners (Cottam et al., 2004; Tatham and Spens, 2011), create internal awareness for cultural differences (Dowty and Wallace, 2010) and adopt more decentralized decision-making (Baharmand et al., 2017).

It is noteworthy that some of the panelists’ recommendations are, at least to a certain extent, conflicting or controversial. For example, experts conform with the literature (Cottam et al., 2004) regarding the fact that close relationships can ease coordination and improve mutual understanding. They also highlight, however, that such relationships can endanger an aid agency’s reputation by causing a perception of favoritism. Similarly, experts recognize that long-term contracts can be beneficial for building partnerships (Maon et al., 2009), but counter that they increase an organization’s dependency and limit its flexibility. Also with respect to performance-based service fees the panel did not reach a consensus. While some argue that it is important to implement them (Balcik et al., 2010), other consider them counterproductive. As mentioned in Section 3.3, the usefulness of selected activities is certainly influenced by contextual factors, such as the value of the service purchase, size and funding structure of an aid agency, the geographic context of the service (before or after point-of-entry) or the phase of disaster relief. This study’s aim was to provide a comprehensive picture on promising activities for organizations engaged in outsourcing. It was an unfortunate, but inherent consequence that we were not able to go into the details of all identified controversies and had to leave some of them unresolved. We suggest that future research provides more clarity regarding the respective matters.

6. Summary and outlook

This study investigated best practices of aid agencies for ensuring the success of outsourcing logistics to commercial LSPs in disaster relief. It is based on a Delphi study with a panel of 31 experienced practitioners and a complementary focus group with 12 experts. As it is one of the first studies in the field to apply the Delphi method, it also examined the application of the technique to the field of humanitarian logistics.

Our research revealed 12 best practices and a compilation of more than 100 activities for putting these practices into action. Experts consider a proper balance between efficiency and compliance, a detailed contract and a detailed service request the most important practices. The practice of balancing efficiency and compliance is a unique addition of our study to literature from the commercial context and results from the influential role of donors in disaster relief. In contrast to existing literature, building trust and establishing win–win situations have a lower importance for practitioners than most of the other best practices. This apparent contradiction is potentially a consequence of the great challenges for building trust in disaster relief. Apart from providing guidance to aid agencies, the developed ranking of best practices can help to prioritize future academic research, since it reflects the priorities of practitioners.

The findings have certain limitations and can be extended in several directions. Even though we involved LSPs in the focus group, our results clearly represent the one-sided perspective of aid agencies. While this was our intended focus, future studies in this context could aim to determine best practices for LSPs as well (Heaslip, 2013; Vega and Roussat, 2015). Moreover, our research design did not allow to investigate potential interdependencies between individual practices, which are likely to exist. Future research could deepen the insights in this regard, for example, by applying interpretive structural modeling as illustrated by Qureshi et al. (2007). In general, future studies could further examine specific aspects of the 12 practices. For instance, they could determine criteria for selecting service providers and indicators for evaluating their performance (Baharmand et al., 2017). It would also be important to better understand the role of donors in the context of outsourcing and to identify approaches for further harmonizing their requirements with the operational environment of aid agencies (Burkart et al., 2016). Likewise, future research could provide more clarity with respect to some of the unresolved controversies in our results, for example, how aid agencies should deal with close relationships or performance-based fees.

Finally, we would like to encourage other researchers to explore the use of the Delphi method for their work. We found it to be a promising approach for assessing the perspectives of humanitarian experts in a structured and confirmable way. While specific limitations exist, we strongly believe that these can be addressed by an adequate research design.

Relative ranking of best practices

Rank ID Best practice Mean rating
 1 P03 Ensure both compliance and efficiency in the procurement process 3.63
 2 P06 Use a detailed written contract as safeguard 3.63
 3 P04 Be detailed in the request for proposal or request for quotation 3.56
 4 P01 Prepare for the engagement of logistics service providers 3.37
 5 P02 Set up and maintain a plan for the engagement of LSPs 3.33
 6 P05 Take actions to ensure the selection of the right provider for the right task 3.33
 7 P08 Avoid unnecessary liabilities and risks 3.30
 8 P09 Run a formal process for performance evaluation 3.19
 9 P10 Build mutual understanding and trust with provider 3.11
10 P12 Work with provider as an integrated team 3.11
11 P11 Design engagement as win–win situation 2.89
12 P07 Set up financial incentives to align goals of service provider 2.85a

Notes: Scale: 1 – not important; 2 – somewhat important; 3 – important; 4 – very important. aNo consensus among panelists (IQR>1.0 and/or rwg⩽0.3)

Exemplary references for best practices in literature on commercial logistics

ID Best practice Exemplary references in literature on commercial logistics
P01 Prepare for the engagement of logistics service providers Clear up-front planning and joint planning (Goles and Chin, 2002; de Grahl, 2011; Hauptmann, 2007; Lambert et al., 1999; Selviaridis et al., 2008)
P02 Set up and maintain a plan for the engagement of logistics service providers Clear up-front planning and joint planning (Goles and Chin, 2002; de Grahl, 2011; Hauptmann, 2007; Lambert et al., 1999; Selviaridis et al., 2008)
P03 Ensure both compliance and efficiency in the procurement process
P04 Be detailed in the request for proposal or request for quotation Alignment of expectations, well-defined requirements and thoroughness in contract negotiations (Frankel et al., 1996; Hofenk et al., 2011; Lambert et al., 1999; van Laarhoven et al., 2000)
P05 Take actions to ensure the selection of the right provider for the right task Importance of working with a compatible provider, e.g. regarding capabilities, reputation and culture (Frankel et al., 1996; Goles and Chin, 2002; Hauptmann, 2007; Lambert et al., 1999; Leahy et al., 1995)
P06 Use a detailed written contract as safeguard Written agreement or contract and clear ground rules (Frankel et al., 1996; Tate, 1996)
P07 Set up financial incentives to align goals of service provider Management of provider opportunism (Hauptmann, 2007; Selviaridis and Norrman, 2015)
P08 Avoid unnecessary liabilities and risks Provisions for adjustments and exit (Leahy et al., 1995; Selviaridis et al., 2008; Tate, 1996)
P09 Run a formal process for performance evaluation Joint operating controls and strong performance orientation (Lambert et al., 1999; Leahy et al., 1995; Qureshi et al., 2007; Selviaridis et al., 2008; van Laarhoven et al., 2000)
P10 Build mutual understanding and trust with provider Relationship with mutual trust and good understanding (Frankel et al., 1996; Goles and Chin, 2002; Lambert et al., 1999; Leahy et al., 1995; Qureshi et al., 2007; Selviaridis et al., 2008; Tate, 1996)
P11 Design engagement as win–win situation Sharing of benefits and risks/mutual benefits, incl. open-book contracts (Goles and Chin, 2002; de Grahl, 2011; Lambert et al., 1999; Leahy et al., 1995; Selviaridis et al., 2008)
P12 Work with provider as an integrated team Close working relationships and open (two-way) communication and information exchange (Frankel et al., 1996; Goles and Chin, 2002; de Grahl, 2011; Lambert et al., 1999; Leahy et al., 1995; Qureshi et al., 2007; Selviaridis et al., 2008; Tate, 1996; van Laarhoven et al., 2000)

Participants of Delphi study and focus group

ID Current job title Exp. in hum. log. (years) Focus of experience (headquarters/field) Participation (e-Delphi/focus group (FG))
Expert 1 (E1) Deputy Head of Procurement and Logistics 14 HQ and field e-Delphi
Expert 2 (E2) Global Logistics Advisor 18 HQ and field e-Delphi
Expert 3 (E3) Administrator Department Of Operations 4 HQ and field e-Delphi and focus group
Expert 4 (E4) Deputy Logistics Director 12 HQ and field e-Delphi
Expert 5 (E5) Head of Supply Chain 10 HQ and field Focus group
Expert 6 (E6) Supply Chain Manager 17 HQ and field e-Delphi and focus group
Expert 7 (E7) Senior Program Officer 15 Field e-Delphi
Expert 8 (E8) Head of Private Sector Partnerships 10 HQ and field Focus group
Expert 9 (E9) Supply Chain Director 3 n/a e-Delphi
Expert 10 (E10) Head of Transport 16 HQ e-Delphi
Expert 11 (E11) Head of Procurement and Logistics 10 HQ and field e-Delphi and focus group
Expert 12 (E12) Manager, Purchasing and Logistics 14 HQ Focus group
Expert 13 (E13) Solution Developm. Mgr in Int. Log. 4 n/a Focus group
Expert 14 (E14) Mgr Private Sector Partnerships 2 n/a Focus group
Expert 15 (E15) Technical Referent Log. Platform 8 Field e-Delphi
Expert 16 (E16) Operational Mgmt. and Logistics Trainer 20 HQ and field e-Delphi and focus group
Expert 17 (E17) Project Manager 13 HQ and field e-Delphi
Expert 18 (E18) Head of Procurement and Logistics 17 HQ and field e-Delphi
Expert 19 (E19) Procurement Officer 7 HQ and field e-Delphi
Expert 20 (E20) Head of Aid and Relief for IMEA 8 n/a Focus group
Expert 21 (E21) Global Supply Chain Mgmt. Officer 4 HQ and field e-Delphi
Expert 22 (E22) Human Rights Officer 7 HQ and field e-Delphi
Expert 23 (E23) Operational Logistics – Head of Service 20 HQ and field e-Delphi
Expert 24 (E24) Director, Supply Chain Management 10 HQ and field e-Delphi
Expert 25 (E25) Senior Advisor for Logistics 25 HQ and field Focus group
Expert 26 (E26) Chief Vendor Management and Supply 25 HQ and field e-Delphi
Expert 27 (E27) Chief of Air Operations 22 HQ e-Delphi
Expert 28 (E28) Global Lead for Emergency Logistics 20 HQ and field e-Delphi
Expert 29 (E29) Program Ass. – Emergency Logistics 2 HQ e-Delphi
Expert 30 (E30) Project Manager 15 HQ e-Delphi
Expert 31 (E31) Finance Advisor 15 Field e-Delphi
Expert 32 (E32) Head of Procurement and Logistics 6 HQ and field e-Delphi
Expert 33 (E33) Senior Supply and Logistics Advisor 12 HQ and field e-Delphi
Expert 34 (E34) Head of Supply and Logistics 13 HQ and field e-Delphi
Expert 35 (E35) Reg. Director for Emergency Programs 15 Field e-Delphi and focus group
Expert 36 (E36) Procurement Coordinator 18 HQ e-Delphi
Expert 37 (E37) Logistics Advisor 4 HQ e-Delphi
Expert 38 (E38) Procurement and Logistics Manager 10 HQ and field e-Delphi

Organizations of participants

Name of organization No. of participants in e-Delphi No. of participants in focus group Country of residency of participants
Caritas International 1 Austria
Danish Refugee Council 4 1 Denmark, South Sudan
Diakonie Disaster Relief 1 Germany
Bill and Melinda Gates Foundation 1 USA
Globale Verantwortung 1 Austria
Handicap International 1 France
ICRC 2 Switzerland
IFRC 1 Switzerland
International Medical Corps 1 1 USA, Croatia
Islamic Relief Worldwide 1 1 Turkey
Lutheran World Relief 1 USA
Médicines Sans Frontières 4 1 France, Switzerland
Oxfam Global 1 UK
Oxfam UK 1 UK
Oxfam Spain 1 Spain
People in Need (PIN) 1 1 Czech Republic
Red Cross Canada 1 Canada
Red Cross Austria 1 1 Austria
Samaritan Austria 1 1 Austria
UN Mission in South Sudan 1 South Sudan
UNHCR 1 Hungary
Welthungerhilfe 1 Germany
WFP/Logistics Cluster 1 1 Italy, Denmark
World Vision Australia 1 Australia
World Vision Germany 1 1 Germany
Damco 1 UK
Kuehne + Nagel 1 UAE

List and rating of activities

Practice ID Activity name Mean rating
P1 1.02 Develop tools and templates (e.g. contract templates, supplier database, performance evaluation checklist) 3.67
P1 1.10 Set up framework agreements (non-binding contracts for longer duration which are ready-to-use in case of need) 3.63
P1 1.08 Perform regular assessments of service provider market (e.g. searching for new providers, market prices, cost structure of providers) 3.56
P1 1.09 Run pre-qualification and pre-selection procedures (e.g. request for information, to have reference list of ready-to-use providers) 3.56
P1 1.03 Train employees on outsourcing (e.g. on tender processes, contracting, vendor management) 3.52
P1 1.06 Exchange best practices and templates with other humanitarian organizations 3.52
P1 1.01 Develop guidelines and standard operating procedures for outsourcing (e.g. process descriptions, approval workflows) 3.50
P1 1.05 Perform regular lessons learnt sessions within team to enable organizational learning 3.46
P1 1.04 Establish internal group of experts for outsourcing of logistics (e.g. in central procurement offices) 3.08
P1 1.07 Engage professional (external) support for outsourcing (e.g. experts for tender process or contract design) 2.76
P2 2.02 Develop exact understanding of service needs and requirements 3.65
P2 2.03 Source multiple alternative providers for same service (e.g. to reduce risks) 3.56
P2 2.07 Perform regular risk analysis and develop contingency plans (e.g. availability of alternative providers) 3.56
P2 2.08 Discuss and plan fallback options (=plan B) with providers (e.g. alternative routes for transportation) 3.48
P2 2.04 Negotiate with providers based on consolidated demands (e.g. across areas or countries) 3.42
P2 2.01 Align plan for engagement of logistics providers with overall logistics strategy 3.35
P2 2.05 Coordinate service purchases with other aid agencies (e.g. alignment of plans to avoid competition or joint tendering of services to increase bargaining power) 3.22
P2 2.06 Plan budget for handling of challenges during engagements 3.08a
P2 2.09 Negotiate prices for potential future service requirements with provider (even if these services are not required right away) 3.00a
P3 3.08 Carefully document every purchase (from request to decision) 3.78
P3 3.02 Allow exceptions from general process for urgent or emergency procurements (with adequate approval workflow) 3.67
P3 3.05 In case of emergencies, send request for proposal only to limited number of providers (minimum three) 3.52
P3 3.07 Involve tender committee to steer procurement for long-term and framework agreements 3.48
P3 3.03 Use open tenders for long-term and framework agreements (i.e. publication of request for tender) 3.46
P3 3.04 In case of one-time services (e.g. single transport order), send request for proposal only to limited number of providers (minimum three) 3.44
P3 3.06 Use direct negotiations with providers (without competitive bidding process) only in cases of very high urgency 3.23a
P3 3.01 Align service procurement process with public procurement regulations and donor requirements 3.11a
P4 4.02 Specify detailed service expectations and requirements when sending request for proposal (e.g. deadlines, technical specifications, documentation requirements) 3.81
P4 4.03 Describe details of bidding process when sending request for proposal (e.g. administrative requirements, timeline) 3.74
P4 4.04 Describe selection and decision process when sending request for proposal (e.g. relevant selection criteria) 3.56
P4 4.01 Highlight context of operation and specifics of humanitarian operation when sending request for proposal (urgency, criticality, etc.) 3.52
P4 4.06 Attach questionnaire to gather further information on provider when sending request for proposal (e.g. company information, experience, customer references) 3.44
P4 4.05 Attach standardized bid template when sending request for proposal (which can be used by providers for making their bid) 3.33
P4 4.07 Attach draft contract when sending request for proposal
P5 5.08 Perform background checks on provider (e.g. regarding independence, legal compliance, working conditions, financial health) 3.70
P5 5.06 Ask providers for track records (credentials) and customer references 3.44
P5 5.09 Perform site visits, physical checks (e.g. of trucks) and field assessments 3.37
P5 5.05 Ask for recommendations within own network (other aid agencies, suppliers, service providers, local authorities) 3.33
P5 5.02 Approach as many potential providers with a request for proposal (or quotation) as possible (minimum three) 3.30
P5 5.03 Leave enough time for service providers to respond to request for proposal (or quotation) 3.26
P5 5.01 Ensure good media support for publication of open tenders 3.15
P5 5.07 Perform bid presentations and questioning of representatives 3.15
P5 5.04 Develop understanding of preferred type of logistics service provider (e.g. rather small or big, international or local, asset-based or non-asset-based providers) 3.04
P5 5.10 Test new providers before full engagement (e.g. in less important operations or with short contract durations) 3.04
P6 6.04 Include service-level agreements and performance indicators into the contract (e.g. deadlines, response times) 3.67
P6 6.08 Describe detailed provider responsibilities in contract (e.g. documentation, reporting) 3.67
P6 6.03 Describe scope of services in detail in contract (e.g. in-scope vs out-of-scope) 3.63
P6 6.09 Describe indemnities, liquidated damages or penalties in contract (e.g. for product damages or theft) 3.59
P6 6.05 Describe structure of service fees and payment terms in contract 3.52
P6 6.06 Include duration of contract and rules for contract extension in contract 3.52
P6 6.11 Include provisions for adjustments and termination of contract 3.44
P6 6.14 Include code-of-conduct and data protection clauses (e.g. non-disclosure agreement) into the contract 3.41
P6 6.12 Clarify process for arbitration of disputes in contract 3.38
P6 6.10 Regulate insurance coverage in contract 3.37
P6 6.13 Clarify legal framework (e.g. court of jurisdiction, governing law) in contract 3.33
P6 6.07 Describe mode of cooperation in contract (e.g. ordering process, planning process) 3.15
P6 6.02 Describe context of operation and specifics of humanitarian mission in contract (e.g. urgency, criticality) 2.85
P6 6.01 For one-time services (e.g. single transport orders), limit contract to most important elements (e.g. by e-mail confirmation) 2.59
P7 7.03 Define adequate documentation as precondition for payments (e.g. provision of waybills) 3.70
P7 7.01 Avoid payments in advance 3.08a
P7 7.05 Design contracts as open-book (i.e. providing full transparency on true cost of provider) 3.04a
P7 7.02 Design service fee as performance based (i.e. decrease of service fee for performance below target) 2.92a
P7 7.04 Offer (automatic) contract extensions for satisfactory performance 2.50a
P8 8.10 Avoid communicating sensitive data to provider (e.g. beneficiary lists, budget) 3.62
P8 8.04 In longer-term contracts, include sufficient provisions for termination of contract (e.g. because of funding issues, end of mission) 3.56
P8 8.06 Obtain written commitment of provider to adhere to agreed service fees (e.g. by using fixed price contracts) 3.56
P8 8.03 In longer-term contracts, avoid exclusivity clauses (i.e. restriction to single service providers) 3.54
P8 8.05 In longer-term contracts, include future price adjustments according to market valuation (e.g. alignment with best published price) 3.38
P8 8.08 Avoid (impression of) too close personal relationships with provider (e.g. because of potential criticism of bribes) 3.37
P8 8.07 Have all contracts reviewed by legal expert 3.35
P8 8.09 Follow-up in written on all oral communication to have proof of conversation 3.30
P8 8.02 In longer-term contracts, avoid volume commitments (e.g. minimum service volume) 3.22
P8 8.01 Set short contract duration (e.g. one-time services) in case of high uncertainty (e.g. economic volatility, chaotic situations, at the beginning of an operation) 2.85a
P9 9.01 Monitor execution of activities by service provider (e.g. status tracking) 3.56
P9 9.02 Perform inspections and audits (e.g. product quality, documentation compliance) 3.56
P9 9.09 Maintain record of performance for each provider 3.50
P9 9.04 Use a limited number of qualitative and quantitative performance indicators 3.35
P9 9.03 Run performance review in regular intervals (e.g. in fixed time intervals or at milestones) 3.30
P9 9.07 Measure provider against benchmarks (e.g. best-performing service provider) 3.23
P9 9.06 Involve provider in definition of performance metrics 3.15
P9 9.08 Involve multiple people into evaluation of service (e.g. recipient of deliveries) 3.11
P9 9.05 Set up performance metrics which measure “humanitarian performance” (no business-as-usual KPIs) 3.04
P10 10.10 Always keep own commitments (e.g. compliance of promised payments) 3.78
P10 10.09 Be flexible if changes to agreed terms are required 3.28
P10 10.11 Share projections (e.g. future service demands) with provider 3.28
P10 10.03 Give feedback to loosing tenderers on rejection decision (e.g. reason of rejection) 3.27
P10 10.05 Review the contract in detail together with the service provider 3.27
P10 10.07 Define relationship manager on both sides (management level) 3.26
P10 10.01 In case of open publication of service request, additionally send direct invite to important providers 3.04a
P10 10.04 Set up contracts with “longer duration” (e.g. multiple months) 3.04a
P10 10.06 Explain specifics of humanitarian context to managerial and operational staff of provider in person 3.04
P10 10.02 Preferentially select providers with which you have experience in working together 2.96a
P10 10.08 Have selected staff of provider working in your organization for some time 2.58a
P11 11.01 Negotiate and fix fair service fees (which allow adequate profits for provider) 3.20
P11 11.02 In longer-term contracts, include future price adjustments based on official cost index (e.g. fuel price index) 3.15a
P11 11.08 Acknowledge good performances of provider (e.g. in letter of reference or letter of acknowledgment) 3.15
P11 11.05 Link service fees to actual costs of service provider (e.g. cost-plus contract) 2.57
P11 11.03 Design service fee as performance based (i.e. increase of service fee for performance above target) 2.44a
P11 11.07 Let provider benefit from publicity (“humanitarian image”) 2.40a
P11 11.10 Make faster payments in case of good performance 2.23a
P11 11.04 Offer (automated) contract extensions for satisfactory performance 2.08a
P11 11.06 Guarantee fixed minimum payment to provider (independent of transaction volume) 2.04a
P11 11.09 Reward service provider team with gifts (e.g. family trips) 1.08
P12 12.03 Communicate frequently, honestly and transparently 3.67
P12 12.01 Define and document contact persons and numbers with 24/7 availability on both sides (incl. backups) 3.50
P12 12.08 Work together to find joint solutions for issues and challenges and support provider on his tasks 3.37
P12 12.05 Proactively share all relevant information with provider 3.33
P12 12.09 Provide continuous performance feedback to provider and conduct regular lessons learnt sessions with provider 3.30
P12 12.10 Involve provider in internal training (e.g. standard operating procedures, best practices) 3.00a
P12 12.02 Involve provider in logistics planning of operation 2.92a
P12 12.04 Work face to face whenever possible 2.92a
P12 12.06 Exchange information with provider through electronic data interchange (EDI) 2.76
P12 12.07 Allow provider access to parts of own IT system (e.g. for warehouse services) 2.21a

Notes: Scale: 1 – not useful; 2 – somewhat useful; 3 – useful; 4 – very useful. aNo consensus among panelists (IQR>1.0 and/or rwg⩽0.3)



Seven participants preferred a phone interview to the online survey. In these cases, we used the online questionnaire as an interview guideline. All interviews took between 45 and 90 min and were transcribed in verbatim form for further analysis.


E19 refers to Expert 19 (see Table A1). Direct quotes from experts are provided without modifications. Any adjustments are highlighted using squared brackets.

Appendix 1. Participants of Delphi panel

Table AI

Table AII

Appendix 2. List and rating of activities

Table AIII

Appendix 3. Questionnaire 1 (shortened)

The first page of the questionnaire introduced the goal of the study, explained the usage of the results, outlined the procedure of the study and described the structure of the questionnaire. The questions were then presented in three consecutive sections:

  • Section I: personal experience

    • Please shortly describe a humanitarian operation in which you engaged commercial logistics service provider(s) (country, aim, reason, year, etc.).

    • What were your role and responsibilities with respect to the engagement of the logistics service provider(s) in this humanitarian operation?

    • Please describe the logistics service providers which you engaged in this humanitarian operation (e.g. “small local organization with limited capacity” or “major international enterprise”)

    • What kind of relationship did your organization have with the service provider(s) before the start of the operation?

    • For how long did you engage the service provider(s) in this humanitarian operation?

    • For which logistics activities did you engage service provider(s) in this humanitarian operation?

    • Why did you engage logistics service provider(s) for these activities in this humanitarian operation?

    • To which extent would you consider the described engagement of logistics service providers successful? Why?

    • What should your colleague do in the same way as you did it during the described operation?

    • Why should your colleague do the above activities in the same way?

    • What could your colleague do differently than you did it during the described operation?

    • What would be the advantages of doing the above activities differently?

  • Section II: humanitarian operations in general

    Looking beyond your past experience and speaking in general: What do you consider most important for successfully engaging logistics service providers in humanitarian operations?

  • Section III: specific recommendations

    Please assume that a colleague of yours wants to engage a logistics service provider under similar conditions as during the operation which you have just described. What advice would you give your colleague based on your experience in the mentioned operation?

    • What should your colleague pay attention to when running the tender process?

    • Which criteria should your colleague apply when choosing between different service providers?

    • For how long should your colleague engage the chosen service provider(s)?

    • How should your colleague design the compensation and payment of the service provider(s)?

    • Which content should your colleague include into the contract with the service provider(s)?

    • What else should your colleague pay attention to when negotiating the compensation and setting up the contract?

    • In which way and how frequently should your colleague coordinate and align with the service provider(s)?

    • Which authorizations and responsibilities should your colleague assign to the service provider?

    • What data and information should your colleague share with the service provider(s) and how should your colleague organize the exchange of data and information?

    • Which criteria should your colleague use to evaluate the performance of the service provider(s)?

    • In which way and how often should your colleague measure the performance of the service provider(s)?

    • Which kind of performance incentives (e.g. rewards and penalties) should your colleague set up for the service provider(s)?

    • How should your colleague handle challenges and difficulties in the collaboration with the logistics service providers(s)?

Appendix 4. Questionnaire 2 (extract)

The questionnaire contained one page for each of 12 practices. Each of the 12 pages was designed in the same way. For illustration we provide the extract from the first page.

Practice 1: Prepare for the engagement of logistics service providers

Practitioners recommend to take actions which prepare the humanitarian organization for the engagement of logistics service providers. The preparation should take place internally (e.g. training) and externally (e.g. framework agreements with preferred providers). Having the right structures in place allows to be more efficient when the need for such services occurs:

  1. How useful do you consider the following activities, which were recommended by practitioners in the context of this practice? (not useful, somewhat useful, useful, very useful):

    • Develop guidelines and standard operating procedures for outsourcing (e.g. process descriptions, approval workflows).

    • Develop tools and templates (e.g. contract templates, supplier database, performance evaluation checklist), etc.

  2. How important do you consider this practice (Prepare for the engagement of logistics service providers) in general? (not important, somewhat important, important, very important).

  3. Do you have any comment with respect to your judgment?


Abidi, H., de Leeuw, S. and Klumpp, M. (2014), “Humanitarian supply chain performance management: a systematic literature review”, Supply Chain Management: An International Journal, Vol. 19 Nos 5/6, pp. 592-608.

Abidi, H., de Leeuw, S. and Klumpp, M. (2015), “The value of fourth-party logistics services in the humanitarian supply chain”, Journal of Humanitarian Logistics and Supply Chain Management, Vol. 5 No. 1, pp. 35-60.

Aengenheyster, S., Cuhls, K., Gerhold, L., Heiskanen-Schüttler, M., Huck, J. and Muszynska, M. (2017), “Real-time Delphi in practice – a comparative analysis of existing software-based tools”, Technological Forecasting and Social Change, Vol. 118, pp. 15-27, available at:

Ager, A., Stark, L., Akesson, B. and Boothby, N. (2010), “Defining best practice in care and protection of children in crisis-affected settings: a Delphi study”, Child Development, Vol. 81 No. 4, pp. 1271-1286.

Akkermans, H.A., Bogerd, P., Yücesan, E. and Van Wassenhove, L.N. (2003), “The impact of ERP on supply chain management: exploratory findings from a European Delphi study”, European Journal of Operational Research, Vol. 146 No. 2, pp. 284-301.

Bagchi, A., Paul, J.A. and Maloni, M. (2011), “Improving bid efficiency for humanitarian food aid procurement”, International Journal of Production Economics, Vol. 134 No. 1, pp. 238-245.

Bagchi, P.K. and Virum, H. (1996), “European logistics alliances: a management model”, The International Journal of Logistics Management, Vol. 7 No. 1, pp. 93-108.

Baharmand, H., Comes, T. and Lauras, M. (2017), “Managing in-country transportation risks in humanitarian supply chains by logistics service providers: insights from the 2015 Nepal earthquake”, International Journal of Disaster Risk Reduction, Vol. 24, pp. 549-559, available at:

Balcik, B., Beamon, B.M., Krejci, C.C., Muramatsu, K.M. and Ramirez, M. (2010), “Coordination in humanitarian relief chains: practices, challenges and opportunities”, International Journal of Production Economics, Vol. 126 No. 1, pp. 22-34.

Bealt, J., Fernández Barrera, J.C. and Mansouri, S.A. (2016), “Collaborative relationships between logistics service providers and humanitarian organizations during disaster relief operations”, Journal of Humanitarian Logistics and Supply Chain Management, Vol. 6 No. 2, pp. 118-144.

Beamon, B.M. and Balcik, B. (2008), “Performance measurement in humanitarian relief chains”, International Journal of Public Sector Management, Vol. 21 No. 1, pp. 4-25.

Binder, A. and Witte, J.M. (2007), “Business engagement in humanitarian relief: key trends and policy implications”, Global Public Policy Institute (GPPi), London, available at: (accessed November 5, 2019).

Bogner, A. and Menz, W. (2009), “The theory-generating expert interview: epistemological interest, forms of knowledge, interaction”, in Bogner, A., Littig, B. and Menz, W. (Eds), Interviewing Experts, Palgrave Macmillan, London, pp. 43-80.

Bogner, A., Littig, B. and Menz, W. (2014), Interviews mit Experten. Qualitative Sozialforschung, Springer, Wiesbaden.

Boyson, S., Corsi, T., Dresner, M. and Rabinovich, E. (1999), “Managing effective third party logistics relationships: what does it take?”, Journal of Business Logistics, Vol. 20 No. 1, pp. 73-100.

Burkart, C., Besiou, M. and Wakolbinger, T. (2016), “The funding – humanitarian supply chain interface”, Surveys in Operations Research and Management Science, Vol. 21 No. 2, pp. 31-45.

Business Dictionary (2018), “What is best practice? Definition and meaning”, available at: (accessed June 6, 2018).

Cornick, P. (2006), “Nitric oxide education survey – use of a Delphi survey to produce guidelines for training neonatal nurses to work with inhaled nitric oxide”, Journal of Neonatal Nursing, Vol. 12 No. 2, pp. 62-68.

Cottam, H., Roe, M. and Challacombe, J. (2004), “Outsourcing of trucking activities by relief organisations”, Journal of Humanitarian Assistance, Vol. 1 No. 1, pp. 1-26, available at:

Cozzolino, A. (2012), Humanitarian Logistics: Cross-Sector Cooperation in Disaster Relief Management, ISBN 364230186X, Springer-Verlag, Berlin and Heidelberg.

Cozzolino, A., Wankowicz, E. and Massaroni, E. (2017), “Logistics service providers’ engagement in disaster relief initiatives: an exploratory analysis”, International Journal of Quality and Service Sciences, Vol. 9 Nos 3/4, pp. 269-291.

Creswell, J.W. (2009), Qualitative Inquiry and Research Design: Choosing Among Five Approaches, SAGE, Thousand Oaks, CA.

Dalkey, N. and Helmer, O. (1963), “An experimental application of the Delphi method to the use of experts”, Management Science, Vol. 9 No. 3, pp. 458-467.

Dalkey, N.C., Rourke, D.L., Lewis, R. and Snyder, D. (Eds) (1972), Studies in the Quality of Life: Delphi and Decision-Making, Lexington Books, Lexington, MA.

Day, J. and Bobeva, M. (2005), “A generic toolkit for the successful management of Delphi studies”, The Electronic Journal of Business Research Methodology, Vol. 3 No. 2, pp. 103-116.

de Grahl, A. (2011), Success Factors in Logistics Outsourcing, ISBN 3834970840, Springer Science and Business Media, Wiesbaden.

Delbecq, A.L., Van de Ven, A.H. and Gustafson, D.H. (1975), Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes, ISBN 0673075915, Scott Foresman, Glenview, IL.

Diamond, I.R., Grant, R.C., Feldman, B.M., Pencharz, P.B., Ling, S.C., Moore, A.M. and Wales, P.W. (2014), “Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies”, Journal of Clinical Epidemiology, Vol. 67 No. 4, pp. 401-409.

Dowty, R.A. and Wallace, W.A. (2010), “Implications of organizational culture for supply chain disruption and restoration”, International Journal of Production Economics, Vol. 126 No. 1, pp. 57-65.

Egan, M.J. (2010), “Private goods and services contracts: increased emergency response capacity or increased vulnerability?”, International Journal of Production Economics, Vol. 126 No. 1, pp. 46-56.

Fereday, J. and Muir-Cochrane, E. (2006), “Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development”, International Journal of Qualitative Methods, Vol. 5 No. 1, pp. 80-92.

Fielding, N. and Fielding, J. (1986), Linking Data, Qualitative Research Series No. 4, Sage, London.

Frankel, R., Schmitz Whipple, J. and Frayer, D.J. (1996), “Formal versus informal contracts: achieving alliance success”, International Journal of Physical Distribution & Logistics Management, Vol. 26 No. 3, pp. 47-63.

Frewer, L.J., Fischer, A.R., Wentholt, M.T., Marvin, H.J., Ooms, B.W., Coles, D. and Rowe, G. (2011), “The use of Delphi methodology in agrifood policy development: some lessons learned”, Technological Forecasting and Social Change, Vol. 78 No. 9, pp. 1514-1525.

Froschauer, U. and Lueger, M. (2003), Das Qualitative Interview, Facultas, Wien.

Glaser, B.G. and Strauss, A.L. (2017), Discovery of Grounded Theory: Strategies for Qualitative Research, ISBN 1351522167, Routledge, New York, NY.

Goles, T. and Chin, W.W. (2002), “Relational exchange theory and is outsourcing: developing a scale to measure relationship factors”, in Hirschheim, R., Heinzl, A. and Dibbern, J. (Eds), Information Systems Outsourcing, Springer, Berlin and Heidelberg, pp. 221-250.

Goodman, C.M. (1987), “The Delphi technique: a critique”, Journal of Advanced Nursing, Vol. 12 No. 6, pp. 729-734.

Gould, S.A. (2003), “How to source logistics services strategically”, Supply Chain Management Review, Vol. 7 No. 5, pp. 48-54, available at:

Hasson, F. and Keeney, S. (2011), “Enhancing rigour in the Delphi technique research”, Technological Forecasting and Social Change, Vol. 78 No. 9, pp. 1695-1704.

Hasson, F., Keeney, S. and McKenna, H. (2000), “Research guidelines for the Delphi survey technique”, Journal of Advanced Nursing, Vol. 32 No. 4, pp. 1008-1015.

Hauptmann, S. (2007), Gestaltung des Outsourcings von Logistikleistungen, ISBN 9783835007864, 1st ed., DUV Deutscher Universitäts-Verlag, Wiesbaden.

Heaslip, G. (2013), “Services operations management and humanitarian logistics”, Journal of Humanitarian Logistics and Supply Chain Management, Vol. 3 No. 1, pp. 37-51.

Hofenk, D., Schipper, R., Semeijn, J. and Gelderman, C. (2011), “The influence of contractual and relational factors on the effectiveness of third party logistics relationships”, Journal of Purchasing and Supply Management, Vol. 17 No. 3, pp. 167-175.

Hsu, C.-C. and Sandford, B.A. (2007), “The Delphi technique: making sense of consensus”, Practical Assessment, Research & Evaluation, Vol. 12 No. 10, pp. 1-8.

Huang, M., Ren, L., Lee, L.H. and Wang, X. (2015), “4PL routing optimization under emergency conditions”, Knowledge-Based Systems, Vol. 89, pp. 126-133, available at:

Jahre, M., Pazirandeh, A. and Van Wassenhove, L.N. (2016), “Defining logistics preparedness: a framework and research agenda”, Journal of Humanitarian Logistics and Supply Chain Management, Vol. 6 No. 3, pp. 372-398.

James, L.R., Demaree, R.G. and Wolf, G. (1993), “rwg: an assessment of within-group interrater agreement”, Journal of Applied Psychology, Vol. 78 No. 2, pp. 306-309.

Kennedy, H.P. (2004), “Enhancing Delphi research: methods and results”, Journal of Advanced Nursing, Vol. 45 No. 5, pp. 504-511.

Kitzinger, J. (1995), “Qualitative research: introducing focus groups”, BMJ: British Medical Journal, Vol. 311 No. 7000, pp. 299-302.

Kovács, G., Spens, K. and Glenn Richey, R. (2009), “Identifying challenges in humanitarian logistics”, International Journal of Physical Distribution & Logistics Management, Vol. 39 No. 6, pp. 506-528.

Kovács, G., Tatham, P. and Larson, P.D. (2012), “What skills are needed to be a humanitarian logistician?”, Journal of Business Logistics, Vol. 33 No. 3, pp. 245-258.

Lambert, D.M., Emmelhainz, M.A. and Gardner, J.T. (1999), “Building successful logistics partnerships”, Journal of Business Logistics, Vol. 20 No. 1, pp. 165-181.

Leahy, S.E., Murphy, P.R. and Poist, R.F. (1995), “Determinants of successful logistical relationships: a third-party provider perspective”, Transportation Journal, Vol. 35 No. 2, pp. 5-13.

LeBreton, J.M. and Senter, J.L. (2008), “Answers to 20 questions about interrater reliability and interrater agreement”, Organizational Research Methods, Vol. 11 No. 4, pp. 815-852.

Leung, S.-O. (2011), “A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point Likert scales”, Journal of Social Service Research, Vol. 37 No. 4, pp. 412-421.

L’Hermitte, C., Tatham, P., Bowles, M. and Brooks, B. (2016), “Developing organisational capabilities to support agility in humanitarian logistics: an exploratory study”, Journal of Humanitarian Logistics and Supply Chain Management, Vol. 6 No. 1, pp. 72-99.

Lincoln, Y.S. and Guba, E.G. (1985), Naturalistic Inquiry, ISBN 0803924313, Vol. 75, Sage Publications, Beverly Hills, CA.

Linstone, H.A. and Turoff, M. (1975), The Delphi Method: Techniques and Applications, Addison-Wesley, Reading, MA.

Lozano, L.M., García-Cueto, E. and Muñiz, J. (2008), “Effect of the number of response categories on the reliability and validity of rating scales”, Methodology, Vol. 4 No. 2, pp. 73-79.

Ludwig, B. (1997), “Predicting the future: have you considered using the Delphi methodology?”, Journal of Extension, Vol. 35 No. 5, pp. 1-4.

McKenna, H.P. (1994), “The Delphi technique: a worthwhile research approach for nursing?”, Journal of Advanced Nursing, Vol. 19 No. 6, pp. 1221-1225.

Maceviciute, E. and Wilson, T.D. (2009), “A Delphi investigation into the research needs in Swedish librarianship”, Information Research: An International Electronic Journal, Vol. 14 No. 4, pp. 1-24.

Macneil, I.R. (1978), “Contracts: adjustment of long-term economic relations under classical, neoclassical, and relational contract law”, Northwestern University Law Review, Vol. 72 No. 6, pp. 854-905.

Maon, F., Lindgreen, A. and Vanhamme, J. (2009), “Developing supply chains in disaster relief operations through cross-sector socially oriented collaborations: a theoretical model”, Supply Chain Management: An International Journal, Vol. 14 No. 2, pp. 149-164.

Mari Ainikki Anttila, U. (2014), “Human security and learning in crisis management”, Journal of Humanitarian Logistics and Supply Chain Management, Vol. 4 No. 1, pp. 82-94.

Mayring, P. (1991), “Qualitative inhaltsanalyse”, in Flick, U. (Ed.), Handbuch qualitative Sozialforschung, ISBN 3-621-27105-8, Psychologie-Verl.-Union, München, pp. 209-213.

Mayring, P. (2000), “Qualitative content analysis (28 paragraphs)”, Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, Vol. 1 No. 2, available at:

Meredith, J. (1993), “Theory building through conceptual methods”, International Journal of Operations & Production Management, Vol. 13 No. 5, pp. 3-11.

Meuser, M. and Nagel, U. (1991), “ExpertInneninterviews – vielfach erprobt, wenig bedacht. Ein Beitrag zur qualitativen Methodendiskussion”, in Garz, D. and Kraimer, K. (Eds), Qualitativ-empirische Sozialforschung, Westdeutscher Verlag, Opladen, pp. 441-468.

Meuser, M. and Nagel, U. (2009), “The expert interview and changes in knowledge production”, in Bogner, A., Littig, B. and Menz, W. (Eds), Interviewing Experts, Palgrave Macmillan, London, pp. 17-42.

Morgan, D.L. (1996), Focus Groups as Qualitative Research, ISBN 1506318827, 2nd ed., Vol. 16, Sage Publications, Thousand Oaks, CA.

Morgan, D.L. and Krueger, R.A. (1997), Focus Group Kit, ISBN 9780761907602, Sage Publications, Thousand Oaks, CA.

Nagurney, A., Yu, M. and Qiang, Q. (2011), “Supply chain network design for critical needs with outsourcing”, Papers in Regional Science, Vol. 90 No. 1, pp. 123-142.

Nowack, M., Endrikat, J. and Guenther, E. (2011), “Review of Delphi-based scenario studies: quality and design considerations”, Technological Forecasting and Social Change, Vol. 78 No. 9, pp. 1603-1615.

Nurmala, N., de Leeuw, S. and Dullaert, W. (2017), “Humanitarian-business partnerships in managing humanitarian logistics”, Supply Chain Forum: An International Journal, Vol. 22 No. 1, pp. 82-94.

Nurmala, N., de Vries, J. and de Leeuw, S. (2018), “Cross-sector humanitarian-business partnerships in managing humanitarian logistics: an empirical verification”, International Journal of Production Research, Vol. 56 No. 21, pp. 6842-6858.

Okoli, C. and Pawlowski, S.D. (2004), “The Delphi method as a research tool: an example, design considerations and applications”, Information & Management, Vol. 42 No. 1, pp. 15-29.

Parente, R. and Anderson-Parente, J. (2011), “A case study of long-term Delphi accuracy”, Technological Forecasting and Social Change, Vol. 78 No. 9, pp. 1705-1711.

Patton, M.Q. (2002), Qualitative Research and Evaluation Methods, Sage Publications, Thousand Oaks, CA.

Paul, J.A. and Wang, X.J. (2015), “Robust optimization for United States department of agriculture food aid bid allocations”, Transportation Research Part E: Logistics and Transportation Review, Vol. 82, pp. 129-146, available at:

Pazirandeh, A. and Herlin, H. (2014), “Unfruitful cooperative purchasing”, Journal of Humanitarian Logistics and Supply Chain Management, Vol. 4 No. 1, pp. 24-42.

Pettit, S. and Beresford, A. (2009), “Critical success factors in the context of humanitarian aid supply chains”, International Journal of Physical Distribution & Logistics Management, Vol. 39 No. 6, pp. 450-468.

Poppo, L. and Zenger, T. (2002), “Do formal contracts and relational governance function as substitutes or complements?”, Strategic Management Journal, Vol. 23 No. 8, pp. 707-725.

Powell, C. (2003), “The Delphi technique: myths and realities”, Journal of Advanced Nursing, Vol. 41 No. 4, pp. 376-382.

Quero, R.A. (2012), “Reframing coordination challenges for public-private partnerships in disaster preparedness”, Procedia – Social and Behavioral Sciences, Vol. 57, pp. 440-447.

Qureshi, M.N., Kumar, D. and Kumar, P. (2007), “Modeling the logistics outsourcing relationship variables to enhance shippers’ productivity and competitiveness in logistical supply chain”, International Journal of Productivity and Performance Management, Vol. 56 No. 8, pp. 689-714.

Raskin, M.S. (1994), “The Delphi study in field instruction revisited: expert consensus on issues and research priorities”, Journal of Social Work Education, Vol. 30 No. 1, pp. 75-89.

Rayens, M.K. and Hahn, E.J. (2000), “Building consensus using the policy Delphi method”, Policy, Politics, & Nursing Practice, Vol. 1 No. 4, pp. 308-315.

Richardson, D.A., Leeuw, S. and Dullaert, W. (2016), “Factors affecting global inventory prepositioning locations in humanitarian operations – a Delphi study”, Journal of Business Logistics, Vol. 37 No. 1, pp. 59-74.

Rodgers, B.L. and Cowles, K.V. (1993), “The qualitative research audit trail: a complex collection of documentation”, Research in Nursing & Health, Vol. 16 No. 3, pp. 219-226.

Rousseau, D.M., Sitkin, S.B., Burt, R.S. and Camerer, C. (1998), “Not so different after all: a cross-discipline view of trust”, Academy of Management Review, Vol. 23 No. 3, pp. 393-404.

Rowe, G. and Wright, G. (2011), “The Delphi technique: past, present, and future prospects – introduction to the special issue”, Technological Forecasting and Social Change, Vol. 78 No. 9, pp. 1487-1490.

Rowe, G., Wright, G. and Bolger, F. (1991), “Delphi: a re-evaluation of research and theory”, Technical Forecasting and Social Change, Vol. 39 No. 3, pp. 235-251.

Sahebi, I.G., Arab, A. and Moghadam, M.R.S. (2017), “Analyzing the barriers to humanitarian supply chain management: a case study of the Tehran Red Crescent Societies”, International Journal of Disaster Risk Reduction, Vol. 24, pp. 232-241, available at:

Sánchez Gil, J.C. and McNeil, S. (2015), “Supply chain outsourcing in response to manmade and natural disasters in Colombia, a humanitarian logistics perspective”, Procedia Engineering, Vol. 107, pp. 110-121.

Schulz, S.F. (2009), Disaster Relief Logistics: Benefits of and Impediments to Cooperation between Humanitarian Organizations, ISBN 978-3-258-07442-9, 1st ed., Vol. 15 Kuehne Foundation Book Series on Logistics, Haupt, Berne.

Schulz, S.F. and Heigh, I. (2009), “Logistics performance management in action within a humanitarian organization”, Management Research News, Vol. 32 No. 11, pp. 1038-1049.

Selviaridis, K. and Norrman, A. (2015), “Performance-based contracting for advanced logistics services: challenges in its adoption, design and management”, International Journal of Physical Distribution & Logistics Management, Vol. 45 No. 6, pp. 592-617.

Selviaridis, K. and Spring, M. (2007), “Third party logistics: a literature review and research agenda”, The International Journal of Logistics Management, Vol. 18 No. 1, pp. 125-150.

Selviaridis, K., Spring, M., Profillidis, V. and Botzoris, G. (2008), “Benefits, risks, selection criteria and success factors for third-party logistics services”, Maritime Economics & Logistics, Vol. 10 No. 4, pp. 380-392.

Silverman, D. (2013), Interpreting Qualitative Data, Sage, Thousand Oaks, CA.

Singer, E. and Couper, M.P. (2017), “Some methodological uses of responses to open questions and other verbatim comments in quantitative surveys”, Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology, Vol. 11 No. 2, pp. 115-134.

Sink, H.L. and Langley, C.J. Jr (1997), “A managerial framework for the acquisition of third-party logistics services”, Journal of Business Logistics, Vol. 18 No. 2, pp. 163-187.

Skulmoski, G.J., Hartman, F.T. and Krahn, J. (2007), “The Delphi method for graduate research”, Journal of Information Technology Education: Research, Vol. 6, pp. 1-21.

Spekman, R.E. and Davis, E.W. (2004), “Risky business: expanding the discussion on risk and the extended enterprise”, International Journal of Physical Distribution & Logistics Management, Vol. 34 No. 5, pp. 414-433.

Stehr, N. (1994), Knowledge Societies, 1st. publ. ed., ISBN 080397891X, Sage Publications, London, available at:

Supply Chain Council (2008), Supply Chain Operations Reference Model, 11 ed., Supply Chain Council, available at:

Tate, K. (1996), “The elements of a successful logistics partnership”, International Journal of Physical Distribution & Logistics Management, Vol. 26 No. 3, pp. 7-13.

Tatham, P. and Kovács, G. (2010), “The application of ‘swift trust’ to humanitarian logistics”, International Journal of Production Economics, Vol. 126 No. 1, pp. 35-45.

Tatham, P. and Spens, K. (2011), “Towards a humanitarian logistics knowledge management system”, Disaster Prevention and Management: An International Journal, Vol. 20 No. 1, pp. 6-26.

Thomas, A. (2003), Humanitarian Logistics: Enabling Disaster Response, Fritz Institute, San Francisco, CA.

Thomas, A. and Fritz, L. (2006), “Disaster Relief, Inc.”, Harvard Business Review, Vol. 84 No. 11, pp. 114-122.

Thompson, M. (2009), “Considering the implication of variations within Delphi research”, Family Practice, Vol. 26 No. 5, pp. 420-424.

Tomasini, R.M. and Van Wassenhove, L.N. (2009), “From preparedness to partnerships: case study research on humanitarian logistics”, International Transactions in Operational Research, Vol. 16 No. 5, pp. 549-559.

Trestrail, J., Paul, J. and Maloni, M. (2009), “Improving bid pricing for humanitarian logistics”, International Journal of Physical Distribution & Logistics Management, Vol. 39 No. 5, pp. 428-441.

Turoff, M. (1970), “The design of a policy Delphi”, Technical Forecasting and Social Change, Vol. 2 No. 2, pp. 149-171.

van de Linde, E. and van der Duin, P. (2011), “The Delphi method as early warning: linking global societal trends to future radicalization and terrorism in the Netherlands”, Technological Forecasting and Social Change, Vol. 78 No. 9, pp. 1557-1564.

van Laarhoven, P., Berglund, M. and Peters, M. (2000), “Third-party logistics in Europe – five years later”, International Journal of Physical Distribution & Logistics Management, Vol. 30 No. 5, pp. 425-442.

Van Wassenhove, L.N. (2006), “Humanitarian aid logistics: supply chain management in high gear”, Journal of the Operational Research Society, Vol. 57 No. 5, pp. 475-489.

Vega, D. and Roussat, C. (2015), “Humanitarian logistics: the role of logistics service providers”, International Journal of Physical Distribution & Logistics Management, Vol. 45 No. 4, pp. 352-375.

Vega, D. and Roussat, C. (2016), “Towards a characterization of humanitarian organizations as logistics service providers”, in Ojala, L., Töyli, J., Solakivi, T., Lorentz, H., Laari, S. and Lehtinen, N. (Eds), NOFOMA 2016 – Proceedings of the 28th Annual Nordic Logistics Research Network Conference, Turku School of Economics, Turku, pp. 636-654.

von der Gracht, H.A. (2012), “Consensus measurement in Delphi studies: review and implications for future quality assurance”, Technological Forecasting and Social Change, Vol. 79 No. 8, pp. 1525-1536.

Wakolbinger, T. and Toyasaki, F. (2018), “Impacts of funding systems on humanitarian operations”, in Christopher, M. and Tatham, P. (Eds), Humanitarian Logistics: Meeting the Challenge of Preparing for and Responding to Disasters, Kogan Page Publishers, London, pp. 41-57.

Wallendorf, M. and Belk, R.W. (1989), “Assessing trustworthiness in naturalistic consumer research”, in Hirschman, E.C. (Ed.), SV – Interpretive Consumer Research, Association for Consumer Research, Provo, UT, pp. 69-84.

Wang, X., Wu, Y., Liang, L. and Huang, Z. (2016), “Service outsourcing and disaster response methods in a relief supply chain”, Annals of Operations Research, Vol. 240 No. 2, pp. 471-487.

Woudenberg, F. (1991), “An evaluation of Delphi”, Technological Forecasting and Social Change, Vol. 40 No. 2, pp. 131-150.

Zyck, S. and Kent, R. (2014), “Humanitarian crises, emergency preparedness and response: the role of business and the private sector (final report)”, Overseas Development Institute, London.


The authors sincerely thank all participants from aid agencies and service providers for their contributions. Without their efforts, this study would not have been possible. This research is funded by the Austrian Science Fund (FWF): Project No. 26015.

Corresponding author

Timo Gossler can be contacted at:

Related articles