An evidence-based urban DRR strategy for informal settlements

Abstract

Purpose

The purpose of this paper is to analyze the evidence-based research strategy (EBRS) used to evaluate eight projects that applied the neighborhood approach for disaster risk reduction (NA-DRR) in informal urban settlements in Colombia, Guatemala, Haiti, Honduras, Jamaica and Peru, between 2012 and 2017.

Design/methodology/approach

The study covers the first five of the seven EBRS stages: first, identify relevant interventions; second, prepare evaluation questions; third, select evidence sources and implement a search strategy; fourth, appraise evidences and identify gaps; fifth, create an evaluation design to include an extensive literature review, followed by a mixed research method with surveys, focus groups and interviews; disaster risk modeling; georeferencing analysis; and engineering inspections. The last two stages: sixth, apply the evidence, and seventh, evaluate the evidence application, will be addressed in a near future.

Findings

Even though the reference to “evidence” is frequent in the DRR field, it is largely based on descriptive processes, anecdotal references, best practices, lessons learned and case studies, and particularly deficient on the subject of informal and precariousness settlements. The evaluation allowed a deep and broad analysis of NA-DRR in urban informal settlements, comparing it with other DRR strategies implemented by different stakeholders in fragile urban settings, assessing the effectiveness and sustainability of the various DRR interventions.

Originality/value

The abundant data, information and knowledge generated will serve as foundation for forthcoming thematic peer-reviewed publications informing evidence-based DRR research, policy and practice, with emphasis on informal and precariousness settlements in particular.

Citation

Sarmiento, J.P., Polak, S. and Sandoval, V. (2019), "An evidence-based urban DRR strategy for informal settlements", Disaster Prevention and Management, Vol. 28 No. 3, pp. 371-385. https://doi.org/10.1108/DPM-08-2018-0263

1. Introduction

Given the abundance of articles, texts and guidelines on disaster risk reduction (DRR) in the literature, one wonders if there has been a rational, rigorous and systematic process to evaluate the effectiveness of DRR practice at all. The profusion of practitioner-driven literature alludes perhaps to an eagerness on the part of the authors to attract the attention of an audience thirsty for DRR knowledge.

Clearly, there are some serious challenges in our understanding of DRR practice effectiveness: an overconfidence in lessons learned and good practices, an unmistakable distancing between the DRR community and the academia, in addition to the common limitations in the available DRR literature. The DRR community has seen the need to respond to an increasing risk construction process without systematically embracing a rigorous evaluation of DRR interventions for design, effectiveness, efficiency and sustainability. Academia, too, has lagged behind; it has not been proactive in proposing, designing and implementing evaluation processes for DRR interventions that respond to the needs, times and priorities of the DRR community. In this paper we propose the adoption of the evidence-based (EB) approach to assess DRR practices. Kvernbekk (2011, p. 532) explains evidence as the degree of effectiveness of strategies or interventions, or, “the truth-value of our claims about such effectiveness” and that its function is to support, confirm or disconfirm, a hypothesis or theory. As Tellings (2017, p. 584) explains, “evidence” is not the same as “proof.” The “Grades of Recommendation, Assessment, Development, and Evaluation” offers five major areas or domains to assess the quality of evidence proposed in the literature: issues with the study design or execution, inconsistency of results, indirectness of evidence, imprecision of results and publication bias (Pandis et al., 2015).

This study is structured in four major sections: after the introduction, Section 2 addresses the concepts of EB practice, EB research and EB and DRR. Section 3 on the EB strategy applied to the existing urban DRR practices in densely populated informal urban settlements in the Latin American and Caribbean (LAC) region, includes research objectives and questions, theoretical framework, evaluation design, findings and limitations. Finally, Section 4 contains the conclusions.

2. Evidence-based concept

The term EB was first reported in the 1980s (Rosenberg and Donald, 1995), but it was not until 1992 when the Evidence-Based Medicine Working Group published an article about EB and medical education in the Journal of the American Medical Association (Montori and Guyatt, 2008; Rahman and Applebaum, 2011). The initial emphasis was given to EB practice and research in medicine. The last two decades have seen an increase in EB trends and initiatives beyond medicine in areas such as nursing psychology, social work, education, policy, management and decision making. Each field adopted the same principles while developing their own approaches. For the purposes of this study, we will focus on EB practice and EB research from the social sciences perspective and apply it to DRR.

Evidence-based practice

The evidence-based practice (EBP) is understood as “the conscientious, explicit, and judicious use of current best evidence in making decisions about individual patients” (Sackett et al., 1996, p. 71). In the case of DRR, the practice has largely preceded academic and research discussions. DRR “is aimed at preventing new and reducing existing disaster risk and managing residual risk, all of which contribute to strengthening resilience and therefore to the achievement of sustainable development” (United Nations, 2016, p. 16). In almost four decades, the field has evolved from a purely reactive approach, initially dominated by disaster response and recovery, to an anticipatory or preparedness mode. This was followed by a corrective risk management strategy which sought to work on a built environment to reduce risk construction and mitigate disaster impacts, and has finally reached a prospective strategy that seeks a future model of safe and sustainable development. Aware of the intricate and complicated problem, the functional concept of resilience has been introduced in the past two decades in the development community, understanding it as “the ability of people, households, communities, countries, and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth” (USAID, 2012, p. 5). As a result of this complexity in DRR practices, we found multiple disciplines and stakeholders intervening in DRR. However, we observed a poor systematization of processes and outputs, as well as deficient evaluation of the effectiveness and sustainability of the outcomes obtained.

As Tellings (2017, p. 582) indicates, “[EBP] is based on the view that the best possible way to guarantee that something works is to scientifically prove that it works […] [EPB] is seen as a foundational principle for professionals to continue to learn and maintain theoretical and practical competencies throughout their careers.” Hence, the EBP becomes a useful strategy for disaster risk management (DRM), allowing us to validate the effectiveness and sustainability of many of the current practices, with a strong focus on aspects of learning and capacity development.

Evidence-based research

Evidence-based research (EBR) is defined as “the use of prior research in a systematic and transparent way to inform a new study so that it is answering questions that matter in a valid, efficient and accessible manner” (Robinson, 2009). A fundamental EBR principle indicates that studies should be based on practice and real interventions, allowing active participation of implementing practitioners in defining the objectives and research questions.

The systematic review constitutes a critical tool for EBR, which can be used to plan, implement and evaluate the practice, to prepare and review hypotheses and research evidence, as well as guide future research. During the 1990s and early 2000s, DRR studies most often displayed analytical superficiality, remained descriptive at best, and did not reach the level of theoretical, indicative or causal analysis (Schlosser, 2006). Authors such as Kazdin (2008) considered it critical to move beyond the association between an intervention and its outcome, looking for ways to acquire knowledge about causation, context and assumptions as it led to understanding pathways to change (theory of change), identifying moderating variables and stressing the importance and utility of qualitative research. Interestingly, the descriptive DRR studies remain the norm today.

Evidence-based and DRR

In a careful review of the existing DRR EB literature, the first step was identifying a list of databases (both scientific and gray) and relevant websites to find articles and publications about DRR research and practice experiences that have used an EB approach. The databases selected were: Google Scholar, Scopus, World of Science and PreventionWeb; Google search engine was used to look for other relevant websites. The second step entailed the definition of inclusion search criteria, screening titles and abstracts with the terms (DRR) or related words such as (risk reduction/disaster resilience), along with the topic of interest (EB) and related words (approach/research/strategy/practice). Articles were excluded if: they were editorials, opinions or commentaries without any substantial evidence independent of the study design; the content was unrelated to the topics, not available on internet, or written in a language other than English (Table I).

We reviewed the webpages of a number of international organizations such as the United Nations Office for Disaster Risk Reduction, United Nations Development Program (UNDP), World Health Organization and UN-Habitat, and the reference lists (cross-referencing) of the articles judged to be relevant to the topic of interest.

In this study, we defined EB approach for DRR research and practice as a “(process).” The selected documents were qualitatively assessed for a combination of research-and-practice experiences in which identifiable “(questions)” had initiated an EB research process, and ideally, this process had also informed practice (i.e. in the form of manuals, guidelines, toolkits and the like).

The literature search identified 248 titles and abstracts, conducted a full-text review of 32 articles, selecting 8 articles for the evidence synthesis – systematic reviews which synthesize the results of multiple original studies. Figure 1 describes the flow of the literature review conducted.

A mixed deductive–inductive process and team discussions guided our selection of documents from the literature review search. Not constructing an EB study ourselves, we adapted the confidence in the evidence from reviews of qualitative research approach, instead of quality appraisal scores, to categorize each of the selected full-text articles according to its merit in reporting a process of EB DRR: formulation of questions to guide evidence reviews; systemic literature review and evidence synthesis; and DRR practice or intervention.

A number of articles urge introduction of EB policy or decision making, while others provide frameworks and recommendations to develop EB research in specific areas of disaster research, but none on DRR practices or interventions in particular. In some cases, EB actions, regulations and policies are discussed alongside new knowledge, technologies and policy frameworks. Others provide EB data without offering details about the processes that informed the research preparation, nor how these EB data informed practice. Some documents were guidelines and toolkits that apparently used an EB approach but they lacked references to the process or they do not use the “(EB)” term (or the like). A few cases had developed EB policies reviews in the past, but none were directly related to DRR.

A limited number of articles contain references to EB DRR concerning a systemic literature review, also called “(evidence-based evaluation).” At best, articles refer to the development of EB policies and practices for DRM but without going into the details of the processes.

3. The EB strategy applied to the existing urban DRR practices in densely populated informal urban settlements in the LAC region

The background

The United States Agency for International Development’s Office of US Foreign Disaster Assistance (USAID/OFDA)’s Neighborhood Approach for DRR (NA-DRR) programming is closely associated with the reconstruction process of the neighborhood of Ravine Pintade – “(KATYE)” project – in Port-au-Prince, Haiti, after the 2010 earthquake. This KATYE project was implemented by USAID/OFDA, international NGOs (CHF and PCI), and the residents, who agreed to clear the rubble, repair houses, construct transitional shelters, stabilize slopes, rehabilitate sanitation and drainage infrastructure and footpaths, and redesign public spaces for better safety, access and disaster risk mitigation. Key features for the project’s success were: extensive community engagement, a highly consultative planning process and a commitment to upgrade all aspects of the neighborhood for the common good, which helped residents re-establish their physical, economic and social structures. Principles similar to NA-DRR were used in two cases that preceded the Haiti experience: the 1999 Bamako, Mali, flash flooding response, and the 2006–2007 shelter-led project carried out in Kabul, Afghanistan, in response to the conflict situation (Sarmiento, Gelman, Herard and Bittner, 2017).

The NA-DRRs designed to find practical and workable solutions for DRR in densely populated informal urban settlements occupied by vulnerable and marginal communities (Sarmiento and Herard, 2015). Thus, the concept of NA-DRR has evolved as an integrated, multi-sectoral analytical framework that centers on the geographic confines of neighborhoods in order to address disaster risks in urban settings, through a participative planning process, and addresses four priority sectors: shelter and settlements; economic recovery and market systems; water, sanitation and hygiene; and natural and technological risks. The participatory process and framework flexibility of the NA-DRR allows it to adapt to each local context and incorporate the priority sectors as appropriate to the needs of the community (USAID/OFDA LAC, 2015).

Since 2012, USAID has been promoting the NA-DRR strategy in different LAC countries (see Table II) and many of the first iteration of projects have already concluded.

Statement of work

In 2017, USAID assigned the task of evaluating the effectiveness and sustainability of the NA-DRR programming in the LAC region to the authors. The goal of this evaluation was to improve USAID’s understanding of the performance and outcomes of the USAID funded urban programs that utilize the NA-DRR (USAID, 2017). The findings of this evaluation will inform future programming decisions and adjustments to ongoing USAID/OFDA urban DRR programming in the LAC region and globally. More broadly, the findings of the evaluation will add to the evidence base of the “(Neighborhood Approach)” as a DRR tool (Sarmiento et al., 2018).

The objectives and questions

Two objectives were defined for this performance evaluation: understanding the effectiveness and the sustainability of the NA-DRR. The statement of work defined a specific set of questions for each objective that informed the design and implementation of this evaluation. The complete text of the questions is included in the Findings section of this paper.

The evaluation included a third objective on the programming strategy itself, including the programming implementation, alliances and national counterparts.

The theoretical framework

The theoretical framework of NA-DRR was built on two pillars. The first, the disaster risk concept, understood as: “The potential loss of life, injury, or destroyed or damaged assets that could occur to a system, society or a community in a specific period of time, determined probabilistically as a function of hazard, exposure, vulnerability and capacity” (United Nations, 2016, p. 14). Blaikie et al. (1994) developed the “(Pressure and Release Model)” to reflect the complexity of the disaster risk concept, illustrating the process of risk creation, beginning with the root causes, followed by the dynamic pressures, expressed through unsafe conditions and resulting in risk generation that could materialize into disaster situations. The second pillar is represented by the high growth of urban informality and precariousness in recent decades, leading to the generation of slums or informal settlements. According to UN-Habitat III (2015, p. 1), informal settlements are “residential areas in which 1) inhabitants have no security of tenure vis-à-vis the land or dwellings they inhabit, with modalities ranging from squatting to informal rental housing; 2) the neighborhoods usually lack, or are cut off from, basic services and city infrastructure; and 3) the housing may not comply with current planning and building regulations, and is often situated in geographically and environmentally hazardous areas.” According to Sandoval and Sarmiento (2018), approximately 924m people lived in informal settlements or slums around the world in 2001, representing 31.6 percent of the world’s urban population. The 2014 UN-Habitat (2016) report indicates that 104.8m now live in informal settlements in LAC (21.1 percent).

Building upon the two pillars described, we recognized specific unsafe conditions in the urban informal settlements under study to identify four resulting trajectories that form the pathways of influence through which action must be taken: secure land occupation; sufficient and resilient livelihoods; robust and resilient ecosystems; and adequate disaster risk and emergency management. In addition, it is important to address the issue of extreme poverty, which influences the secure land occupation and sufficient and resilient neighborhoods (Sarmiento et al., 2018).

The evaluation design

We defined seven stages for the EB strategy (EBS) of the evaluation methodology: (1) identify relevant interventions; (2) prepare evaluation questions; (3) select evidence sources and implement a search strategy; (4) appraise evidence and identify gaps; (5) create and implement evaluation design; (6) apply the evidence, and (7) evaluate the evidence application. The present study covers the first five EBS stages, beginning with the first two through the definition of two objectives and specific questions related to the effectiveness and the sustainability of the NA-DRR. For stages (3)–(5), the authors conducted an extensive literature review, followed by a mixed research method, including surveys, focus groups and interviews; disaster risk modeling; georeferencing analysis; and engineering inspections.

The research design incorporated several theoretical perspectives for research and state-of-the art technologies to address USAID’s evaluation questions on project effectiveness and sustainability. We defined units of measurement to build a comparable project baseline and assess and value the project outputs and outcomes. We began with an extensive literature review, followed by mixed research methods, including qualitative and quantitative approaches:

• literature review on DRR approaches and interventions implemented in the LAC region;

• seismic risk modeling for the eight projects selected, landslide risk modeling for four projects, and tropical cyclone risk modeling for one project;

• georeferencing and urban pattern analysis for the eight projects selected; and

• site visits and engineering inspections for key physical and environmental interventions in the eight projects selected.

In order to gather primary data from the selected projects, surveys, focus groups and interviews were conducted following an IRB-approved research protocol. These techniques were conducted in the eight neighborhoods across the six different countries where the DRR projects were implemented. The study involved both males and females, and the subjects of the surveys, interviews and focus groups were adults (aged 18 and over with no upper age limitation).

Surveys

The targeted subject for each survey was the head of a household or spouse in the selected neighborhood. We estimated that an average of 100 households per project benefitted directly from the NA-DRR projects, with a total population size of 800 households. The estimated size of the sample was estimated using the sample size calculator, Raosoft Inc., with a margin of error of 4.25 percent and a 95% confidence level, resulting in n=320. Subsequently, the sample was distributed in proportion with the estimated population in the selected neighborhoods. The survey process was completed with an average of 44 surveys per project, and a total of 349 surveys. The survey consisted of a total of 39 questions grouped into four sub-topics: social cohesion, DRR, urban informality and life satisfaction analysis (LSA). A preceding section on demographics and housing conditions collected information on household identification, demographic and socioeconomic aspects of household respondents and housing structural details.

Focus group discussions

We conducted one focus group discussion for each of the eight NA-DRR projects. Focus groups for each neighborhood included approximately 8–12 subjects comprised of community leaders, women, people with disabilities and elders who lived in the selected neighborhood.

Interviews

The research team conducted eight to ten semi-structured interviews per DRR project with local and national government officers, civil society actors, academics and private sector partners. A total of 105 informants participated in the interviews. Interviewees were approached using the snowball stratified sampling technique.

Online Survey

An online survey was geared toward USAID officers in OFDA (Washington, DC, and LAC regional office) with the support of USAID Monitoring and Evaluation.

We used triangulation, an integrative strategy that purposefully uses multiple theoretical positions to analyze data. In this evaluation, we used several qualitative and quantitative methods (see evaluation design) to analyze the data from the eight urban DRR projects. These mixed methods helped to increase the validity of the evaluation and research findings to answer and satisfactorily address USAID’s questions.

Evaluation findings

Although the eight projects evaluated shared NA-DRR characteristics, each project was unique and designed to respond to community needs and distinct socioeconomic and cultural features, thus framing each in specific realities and contexts. Following is a summary of the findings of the questions posed by USAID to guide the evaluation and part of the authors’ report to the funding agency, USAID (Sarmiento et al., 2018). The first four questions respond to the objective of effectiveness and the last three concern project sustainability:

1. To what extent have projects implemented under a NA contributed to reducing community disaster hazard risks in targeted urban communities in the selected projects?

   Four trajectories or pathways of influence were used to reduce community disaster risk:

   • NA-DRR interventions and features associated with secure land occupation: two key interventions illustrate a successful approach to secure land occupation: the land tenure initiative implemented in Portmore; and relocating at-risk communities in Tegucigalpa.

   • Sufficient and resilient livelihoods: two main NA-DRR initiatives demonstrate effective DRR: the small business approach carried out in Medellín; and the network of pulperías (grocery stores) implemented in Tegucigalpa.

   • Robust and resilient ecosystems: three different NA-DRR projects in Lima implemented afforestation projects, initially designed to reduce the risk of rocks falling from slopes and to recover the fragile ecosystem lost over the past decades.

   • Adequate disaster risk and emergency management: physical works such as pathways, access roads, retaining walls and drainage systems are the axes of risk reduction in neighborhood projects. Pathways were common to the six projects in Central and South America as all the settlements are located on steep slopes. Retaining walls were designed and built in a variety of shapes and sizes to protect against landslides. Infrastructure such as channels to manage the runoff in Port-de-Paix, and gabions in Anse-á-Foleur proved to be highly effective during the passage of Hurricane Irma in 2017. Drainage systems ranged in magnitude from small works in the projects in Lima, Medellin and Mixco, to more complex systems, such as the one built in the Tegucigalpa project.

1. Which aspects of the urban NA-DRR are most effective? Which aspects of the urban NA-DRR are least effective?

   To address this question, we used two different approaches: LSA to measure the level of well-being attributed to the NA-DRR projects’ interventions and cost-benefit analysis (CBA) to calculate and compare benefits and costs of the specific NA-DRR interventions selected. The LSA showed that the categories with the highest impact on life satisfaction improvement were physical works and gains in social mobilization. Neighborhoods that received a community empowerment intervention (social mobilization category) increased their life satisfaction by 0.65 points. Considering that on average, the life satisfaction of all neighborhoods in the study was 2.46, the community empowerment intervention produced an increase in life satisfaction of nearly 27 percent. Other categories with interventions that notably impacted life satisfaction were livelihoods and financial mechanisms, and institutional arrangements. The CBA of the DRR interventions revealed that overall, the USAID project interventions had cost-benefit ratios (BCRs) greater than one point, with the access paths being the most cost-beneficial. A BCR of one indicates that the discounted benefit of implementing an intervention equals its cost. The BCR of physical interventions such as access paths ranged from 6.48 in Rímac to 12.16 in Medellin. Using an average value of the statistical cost of life, the BCR for access paths increases to 98.9 and 47.43, respectively, for Medellin and Rímac. The drainage canal in Port-de-Paix, yielded a BCR of 13.19, valued for benefits from avoided loss of household assets and increase in productive business days. Sanitation interventions, such as septic tanks in Mixco, obtained a BCR of 1.62. Benefits were projected for certain interventions, such as the land tenure registration effort in Portmore, for which the target has not yet been met.

2. To what extent is the NA effective as compared to more traditional DRR approaches in LAC?

   To evaluate the effectiveness of the NA-DRR compared to traditional DRR, we used the approach criterion as the center of analysis, since this is the strategy used to address DRR. We identified six DRR approaches – here called categories – to conduct a thorough comparative analysis of the NA-DRR with other traditional initiatives: area-based; market-based; system-based; institutional-based; individual/household-based; and operational. Some of the DRR initiatives fall into more than one category. The NA-DRR promoted by USAID can be primarily classified as area-based, but it further incorporates aspects of other DRR criteria (market-based, system-based, institutional-based, individual/household-based and operational-based). In addition, the concept of “(neighborhood)” used in the NA-DRR goes beyond the pure geographic meaning of the “(area-based)” category: the neighborhood is a living fabric of social, economic and physical features that provides the residents of a particular territory with an identity, a sense of security, safety, and familiarity. Our literature review revealed that institutions such as USAID, OXFAM, UNDP, DFID and the World Bank used area-based approaches to a certain extent, but without emphasis on precariousness, informality and risk exposure.

3. What factors influence the effectiveness (or lack thereof) of urban DRR programs using the NA in each country of focus?

   We considered two categories of influencing factors for the effectiveness of urban DRR programs using the NA-DRR: reflecting on internal aspects of each project and their immediate environment, and referring to the economic, political and social contexts in a broader sense, that is, outside the project’s control. For instance, in the three projects in Lima, we identified several emergencies triggered by “(El Niño)” in 2017 in northern Peru that created a “(window of opportunity)” to introduce innovative DRR practices at different government levels. We also observed that local governments with a greater capacity in urban development avoided silos, fostered cross-sectorial integration and tended to mainstream DRR practices within urban development. This was particularly effective and a common feature in Carabayllo, Medellin, Mixco and Tegucigalpa. Other external factors included the volatile political context in Mixco; turnover of municipal personnel in Lima; organized crime and violence in Medellín; and specific land tenure issues observed in Portmore.

4. To what extent are communities able to integrate DRR practices and take ownership of the NA? What barriers to utilization of the NA exist?

   We developed a community involvement indicator (Table III), using qualitative analysis of focus groups and interviews, to assess four aspects of community involvement: active involvement in planning; allocation of human and financial resources; active involvement in maintenance; and social control.

   In general terms, the communities were able to integrate DRR practices, although only in few cases took ownership of the NA as a whole. With significant differences among countries, neighbors in Mixco, Medellín, Tegucigalpa, and the three projects in Lima demonstrated appropriation of DRR practices such as better garbage and waste water management for reducing flood impacts and afforestation and gardening to stem the risk of landslides and rockslides. In some cases, such as Medellín and Mixco, people achieved a certain level of empowerment as they started to demand more attention and action from local authorities.

5. To what extent are municipal and national authorities incorporating and institutionalizing the urban NA? What evidence (including, but not limited to, policy or urban planning changes) is there that municipal or national authorities are managing urban risk differently due to USAID/OFDA’s urban NA-DRR?

   A local government involvement indicator was developed (Table IV), using qualitative analysis of interviews and field observations to assess four aspects of local government involvement: active involvement in planning; allocation of human and financial resources; active involvement in maintenance; and regulatory action.

   In cases like Carabayllo, Independencia, Mixco, Medellín and Tegucigalpa, the municipalities incorporated new practices, such as the use of GIS and social media for DRR; participatory design and execution of physical works; inter-sectorial working groups for neighborhood development; and inclusion of DRR measures within municipal budget plans. According to our field observations and interviews, the best institutional ownership was achieved in Tegucigalpa, Mixco and Medellín, primarily due to three factors: the level of municipal autonomy to apply DRR; implementers succeeded in creating inter-institutional and inter-sectorial (including private sector) articulations based on agreements and communication, and then translated these into action; and the willingness and commitment of key actors at the highest level of municipal government, such as mayors or municipal managers. On the other hand, participants from Rímac, Portmore and Haiti pointed out factors that limit the institutionalization of the NA-DRR, including: personnel turnover in municipalities; a lack of willingness and commitment from local authorities; and implementer’s lack of capacity/experience in involving local authorities.

6. What enabling factors and factors impeding success contribute to sustainability of the urban NA-DRR? How sustainable could the targeted urban DRR programs be without external donor support?

   Five categories were defined to address the sustainability of the NA projects: social mobilization, institutional arrangements, physical works, environmental improvements and financial mechanisms. Each of these categories comprises both enabling factors and factors that hinder success. Beyond the enabling and impeding factors mentioned, the study found a significant and positive circumstance that we call concatenation. Concatenation refers to the capacity of a project to advance on the achievements of previous projects or initiatives and in turn influence other projects or initiatives to build on its own outputs or outcomes. Through the years of implementation and beyond, these urban DRR projects have shown a remarkable capacity for concatenation. The NA-DRR project in Anse-à-Foleur offers a good example of this phenomenon. The NA-DRR project provided an excellent quality pipeline from the source of the water to the town. Subsequently, the World Bank built ten water tanks, followed by the municipality which built the distribution network. Another example comes from the NA-DRR in Tegucigalpa, where a public university in Honduras took advantage of JICA-sponsored geological studies to advance the studies required by the USAID-sponsored NA-DRR project. At the same time, the NA-DRR project prepared digital elevation mapping based on LiDAR technology (a detection system that uses light from a laser), which now serves the municipality and other projects supported by the international community.

Beyond the characteristics that define the NA-DRR, such as geographic focus, active participation and sectoral concentration, the NA-DRR program strategy has been characterized by closer cooperation among USAID, implementers, beneficiary communities and local and national governments.

Additionally, the introduction of techniques such as systematization (Sarmiento and Herard, 2015) and the post-project review (Sarmiento, Gelman, Jordão and Bittner, 2017) stressed the importance of processes, and a closer follow-up to project implementation, with special attention to the long-term impacts and the outcomes’ sustainability. USAID has fostered the exchange of practices and experiences among implementers, which has resulted in a substantial collective learning process, one that is unique in terms of depth and quality.

Limitations

Evaluating projects after project closeout was a significant challenge. Not all of the actors who had been involved could be located; community members in informal and precariousness settlements are highly mobile. Another important limitation was the collection of information related to designs, technical specifications and budgets of the project interventions that had concluded between one and three years earlier. Likewise, the high turnover of public employees was evident along with the consequent unfamiliarity of the NA-DRR project – the reason why we resorted to interviewing former employers or employees who had changed their position within the same organization for interviews and information gathering.

In relation to the EB evaluation process, there was some level of resistance among administrators and DRR project implementers from the public, private and civil society organizations to contribute to the evaluation research. Conducting trials or tests to verify the effectiveness and sustainability of practices and processes could be seen as a waste of time and resources, with a strong predominance of outputs vs outcomes. Additionally, some individuals may have been reluctant to submit to any testing or surveys that may point out lack of professional knowledge.

4. Conclusion

Institutions and organizations shoulder the onerous task of evaluating DRR projects and have done so under immense time and resource constraints. The research on evaluation has thus been left wanting in the exploration of causal factors of risk. As a result, the DRR community struggles to demonstrate whether interventions successfully fill the gap and address the root problems of risk. One contributing reason for the research gap has been the lack of support from academia. This paper is a call for action aimed at the academic community to support robust evaluation of DRR practices. We propose an EB approach which zeroes in on the causal factors, steers the focus on to initial and conceptual stages of projects, and prioritizes actions for effectiveness, efficiency and sustainability. The EB approach, along with academic support, can help practitioners and DRR actors on the ground, and inform better DRR policy and practices.

The EB evaluation design in this study has allowed us to answer the seven USAID questions while addressing the proposed objectives of effectiveness and sustainability. The multiple DRR practices/interventions observed were analyzed from different perspectives and methodologies. Our evaluation process carefully documented all DRR interventions for study, and demonstrated the effectiveness and sustainability of interventions of those that had sufficient technical support behind them.

In addition to the EB evaluation process and its results, it was clear that the NA-DRR has expanded the consideration/concept of DRR interventions beyond individuals and households to a settlement approach, addressing critical disaster risk drivers and development gaps, and encouraging a long-term vision. USAID revised the terms of reference of its NA-DRR request for proposals (APS), based on the systematization of experiences, project monitoring, and the permanent feedback from USAID/OFDA project officers and implementing partners.

Our study showed the need to balance physical and social interventions to match individual and collective needs and expectations associated with the common good. Protecting the neighborhood and supporting its cohesion and self-determination are important strategies to build community resilience.

The NA-DRR projects strongly contributed to social mobilization easing to collectively overcome obstacles such as poverty, marginalization, insecurity and despair, challenges experienced daily by communities in informal settlements. This study shows a broader scope than the one initially foreseen for the NA, identifying different strategies with strong incidence on DRR but that can stand alone, such as land tenure, rain and storm-water management, afforestation and housing relocation, among others.

The use of state-of-the-art technologies and the exploration and definition of units of measurement were essential to answering the questions proposed by USAID and mark the beginning of a second study phase – the preparation of a series of peer-reviewed publications that will serve to build a catalog of EB DRR practices. This catalog will inform decision makers and practitioners so that they can incorporate this new knowledge into the project cycle and regular DRR practices. Subsequently, outputs and outcomes obtained from those revised practices should feed rigorous evaluation processes that ensure continuous improvement.

Figures

Figure 1

Flow of the literature search

Table I

Literature Search Strategy

Source		Search strategy
1.	Google Scholar (GS)	Publication Year Costume Range 2000–2018. Titles and Abstracts sorted by date: “Disaster Risk Reduction” AND “Evidence-based” (13) “Disaster Risk” AND “Evidence-based” (15) “Disaster Resilience” AND “Evidence-based” (4) “Disaster Risk Reduction” AND “Systematic review” (8)
2.	Scopus (S)	TITLE-ABS-KEY Timespan=2000–2018 “Disaster Risk Reduction” AND “Evidence-based” (25) “Disaster Risk” AND “Evidence-based” (39) “Disaster Resilience” AND “Evidence-based” (12) “Disaster Risk Reduction” AND “Systematic review” (8)
3.	World of Science (v.5.28) (WS)	Indexes=SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, BKCI-S, ESCI Timespan=2000–2018 TS=“Disaster Risk Reduction*” AND “Evidence-based*” (12) TI=“Disaster Risk*” AND TS=“Evidence-based*” (5) TS=“Disaster Resilience*” AND “Evidence-based*” (28) TS=“Disaster Risk Reduction*” AND EXACT “Systemic review” (6)
4.	PreventionWeb (PW)	Filtered by: Documents and Publications; UN and Intl; Acad and Scientific; including attachments Timespan: 2000–2018 EXACT “Disaster Risk Reduction” AND Evidence-based (10) EXACT “Disaster Risk” AND Evidence-based (10) EXACT “Disaster Resilience” AND Evidence-based (5) EXACT “Disaster Risk Reduction” AND Systemic review (6)
5.	Google (G)	Allintitle; Timespan: 2000–2018; Any region “Disaster risk reduction” AND evidence-based (12) “Disaster risk” AND evidence-based (17) “Disaster resilience” AND evidence-based (1) “Disaster risk reduction” AND Systematic review (9)

Table II

Neighborhood approach projects assessed

Country	Peru	Peru	Peru	Colombia	Guatemala	Haiti	Jamaica	Honduras
City	Carabayllo	Independencia	Rimac	Medellin	Mixco	Port-de-Paix	Portmore	Tegucigalpa
Award no.	AID-OFDA-A-14-00024	AID-OFDA-A-14-00025	AID-OFDA-A-14 00023	AID-OFDA-A-14-00026	AID-OFDA-A-12-00013	AID-OFDA-A-12-00012	APS-OFDA-A-14-00027	AID-OFDA-A-13-00023
Implementing partner	Save the Children/US (SC)	PREDES	COOPI	Global Communities (GC)	Project Concern International (PCI)	World Concern Development Organization (WCDO)	Habitat for Humanity (HfH)	GOAL
Dates	October 1, 2014–September 30, 2017	October 1, 2014–March 31, 2017	September 15, 2014–September 14, 2017	October 1, 2014–January 31, 2017	September 28, 2012– September 30, 2016	September 6, 2012–March 31, 2014	September 2014–April 2018	September 23, 2013–December 23, 2016
NA area – hectares	53.4	11.2	48.5	95.7	8.1	46.4	110.4	59.3
NA built area – hectares	42.6	8.1	44.8	131.0	4.8	14.1	104.3	53.4
House’s surface area sqm	127.5	119.3	106.3	67.8	122.9	337.14	75.3	100.8
Total households	3,338	678	4,214	19,333	389	419	13,854	5,299
Household members average	5	5	5	6	6	7	5	5
Total individuals	15,623	3,295	20,987	115,998	2,298	2,780	62,068	27,717
Population density people/hectare	293	295	433	1,211	285	60	562	467
sqm per person	27.24	24.54	21.34	11.29	20.84	50.85	16.80	19.28
NA area – hectares	53.4	11.2	48.5	95.7	8.1	46.4	110.4	59.3
Budget	$1,894,843	$1,303,302	$1,012,662	$1,708,726	$3,082,151	$1,608,992	$1,688,000	$1,377,444

Table III

Community involvement indicator

	Peru	Peru	Peru	Colombia	Guatemala	Haiti	Jamaica	Honduras
City	Carabayllo	Independencia	Rimac	Medellin	Mixco	Port-de-Paix	Portmore	Tegucigalpa
Active involvement in planning (0–25)	15	20	15	20	22.5	5	10	15
Allocation of resources (0–25)	20	15	20	20	22.5	20	15	20
Active involvement in maintenance (0–25)	15	22.5	15	22	17.5	15	15	15
Social control (0–25)	5	0	2.5	2.5	5.0	0	2.5	20
Community involvement (0–100)	55.0	57.5	52.5	64.5	67.5	40.0	42.5	70.0

Table IV

Local government involvement indicator

	Peru	Peru	Peru	Colombia	Guatemala	Haiti	Jamaica	Honduras
City	Carabayllo	Independencia	Rimac	Medellin	Mixco	Port-de-Paix	Portmore	Tegucigalpa
Active involvement in planning (0–25)	10	15	7.5	20	15	5	7	20
Allocation of resources (0–25)	20	10	10	20	17	10	15	20
Active involvement in maintenance (0–25)	15	12.5	15	17.5	20	8	15	22.5
Social control (0–25)	17.5	5	10	12	15	2.5	2.5	6
Local government involvement (0–100)	25.5	62.5	42.5	42.5	67.0	69.5	39.5	68.5