A quasi-experimental method for testing rural design support within a DRM framework

Abstract

Purpose

The purpose of this paper is to describe the quasi-experiment setting to test the formulated design support (developed in the author’s PhD research) within a design research methodology (DRM) framework. This support intends to help designers to consciously engage rural families within a development aid scenario and increase their self-reliance towards their built environment (housing).

Design/methodology/approach

This paper elaborates on the setting in which the design support was tested within a quasi-experiment. The literature section describes the challenges in design research and why the DRM is suitable for this type of empirical research.

Findings

Findings of the paper include a workable setting to organise and evaluate the impact of a design support within a DRM framework on a vulnerable rural community.

Research limitations/implications

The main limitation of the research lies in the study population. Due to financial and time constraints, only four teams could participate in the experiment conducted in rural Kenya.

Social implications

Further research will need to prove that the support works in comparable situations on the African continent.

Originality/value

The quasi-experiment setting within a DRM framework could benefit researchers in comparable empirical investigations.

Citation

Smits, M.W.M. (2019), "A quasi-experimental method for testing rural design support within a DRM framework", Smart and Sustainable Built Environment, Vol. 8 No. 2, pp. 150-187. https://doi.org/10.1108/SASBE-11-2017-0067

1. Introduction

The author’s overall research is situated on Mt Elgon, rural Kenya. As the communities at this location have various income levels and living standards, struggling to achieve an acceptable quality of life (Skevington et al., 2004), it proves an exemplary situation for comparable developing societies on the continent. Based on previously performed research (literature review, local surveys, etc.) and subsequent publications (Smits, 2014, 2017), the overall problem statement underpins that the self-reliance of the majority of rural impoverished inhabitants is decreasing due to the externally introduced building materials and technologies.

The survey results show that most inhabitants still build traditional housing by themselves although they state that traditional housing is not desirable. Rural inhabitants on Mt Elgon wish for a better and more modern habitation’ however, they lack the resources, tools and knowledge to build an improved one without professional – external – support. Moreover, currently involved professionals (designers, engineers, aid workers, etc.) insufficiently include inhabitants in developing processes. Existing inhabitant capacities[1] are neither analysed nor incorporated in the decision-making. As a result, professionals fail to transfer the applied knowledge in articulating building solutions.

As an effect, inhabitants cannot maintain realized housing, they cannot build new housing without external professional support (financial, organisational, etc.) and consequently lower their level of self-reliance in relation to their built environment. Therefore, professionals require design support that would need to addresses the following three main success criteria to sustain and increase inhabitant self-reliance towards their built environment:

1. increasing the level of inhabitant participation in establishing their built environment;

2. increase the level of including inhabitant capacities in decision-making concerning their built environment; and

3. effective knowledge transfer between professional and inhabitants.

Existing design tools fail to address all three criteria in one tool; furthermore, there is currently no methodology to assess rural inhabitant capacities in relation to their built environment. Therefore, the author previously described a concept support tool[2] enabling professionals to advise impoverished rural communities how they can build and improve houses by themselves. However, to test the impact and overall effectiveness of the design support is equally important. Due to the specificity of the design support and the context in which it is being tested, there are no direct applicable experiments designs available. Therefore, the aim for this paper is to design a framework for the experiment in which the support tool can be tested and evaluated.

Consequently, the next section describes the identified methodology (design research methodology (DRM)) and the subsequent framework; Section 3 succinctly introduces the articulated support followed by the section describing the quasi-experiment framework in which the support was tested. Section 5 describes how the support was tested within a quasi-experiment, followed by the section explaining how the impact was measured.

The last section of the paper projects the conclusions, indicating how the formulated quasi-experiment design will provide researchers working in comparable empirical research with a suitable framework. The framework and the experiment design are unique in their design and application (in a vulnerable context); moreover, offer an applicable framework for the increasing demand from humanitarian organisations. Positioning the results in an applied context is highly complicated (variables) and requires a separate publication to elaborate in length. However, the implications of the results have been included in the conclusions of this paper.

2. Methodology and framework

The author’s research is an empirical investigation of the explanation, prediction and prescription (Weggeman, 2001) of how current design and building methodologies applied by professionals affect rural inhabitant’s self-reliance in relation to their built environment, aiming at developing a functioning design support tool for professionals wanting to design and realise houses in the Mt Elgon area. In comparison to other methodologies, this research focusses on implementing, measuring and altering the methodology in practice, describing suitable support, providing clear insights in its functioning and making improvements to it.

Design research is a rapidly growing research domain (Joost et al., 2016) and specifically, the role of design research and how it constitutes to articulating solutions in practice. Many investigate the importance of performing design research (Birkhofer, 2011; de Vries et al., 2013; Rampino, 2012) or advocate the implementation of the design research in practice (Crouch and Pearce, 2013; Laurel, 2003; Rampino, 2012). Only few research works have studied the practical implications of design research within the context of structuring and writing a dissertation (Durdella, 2018). Moreover, current methodologies lack a description of a fundamental framework in which various methods can be deployed according to the investigated phenomenon in practice. Furthermore, they are often industry specific in their approach and application, which might not work in other fields of design.

With architecture engaging in a variety of spatial, social and contextual issues (Awan et al., 2013), researchers in this field are left with the complexity of making the myriad of individual research outcomes comparable. According to Blessing and Chakrabarti (2009), the current design research gives rise to three main issues:

1. the lack of overview of existing research;

2. the lack of use of results in practice; and

3. the lack of scientific rigour.

In their DRM, Blessing and Chakrabarti (2009) developed a framework specifically for the design industry. In their opinion, there were calls for DRM (Cross et al., 1992; Fulcher and Hills, 1996; Reich, 1995); however, the status of design research into its own methodology is poor (Blessing and Chakrabarti, 2009). The formulated process supports the researcher in developing his design research step by step. This framework specifically aims at validating design methodology research by defining goals clearly, describing success criteria, developing strategies, measuring results in practice and measuring and proving success criteria in clear steps.

The DRM depicts a layout that permits the combination of different architectural research methodologies in one framework: how the problem addressed in situ can be evaluated (Mt Elgon, Kenya), how the solution (success) can be articulated by developing a functioning support tool and how the impact of the support can be evaluated. The main goal of the DRM is to describe a suitable support tool or in case of (partial) failure, providing clear insights for possible continuation of the research in the consecutive DRM stages or the research followed up by others.

In Figure 1, the cycle of the DRM framework is shown. Every phase of the design research can be based on deploying mixed methods. The research clarification (RC) uses literature review, expert interviews and case studies (Mt Elgon) to prove the lack in practice of inhabitant’s self-reliance in relation to their built environment. This helps to identify the research problem, goals and criteria. In the Descriptive Study 1, literature review and survey research (Fowler, 2013) identifies the measurable success criteria that potentially improve inhabitant self-reliance towards their built environment.

In the Prescriptive Study 1 (PS1), the success criteria are articulated in an applicable design support. This phase also describes an evaluation model for the impact of the support and how the support was tested within a quasi-experiment setting in situ, which is the part elaborated in this paper. In the last phase, Descriptive Study 2 (DS2) of the basic DRM framework, the impact of the support is evaluated and adjustments to the support described. The PS1 phase is the part of the author’s overall research described in this paper, excluding the development of the support tool. Below, a brief overview of the support is given followed by the description of the quasi-experiments’ setup.

3. The design support guidance

The support was written for architects and engineers who aim at helping Kenyan rural impoverished inhabitants to establish self-reliance in their housing. This first version of the support was developed for local testing in Kenya; however, it is intended to be expanded to fit wider application on the continent. The support was written based on the literature review and expert interviews. This section describes the chapters of the book and mixed methods used in the support; how the support collects data; and how users evaluate its effectiveness in the field.

As the support is applicable to vulnerable inhabitants and communities (culturally, socially and financially), the first chapter of the support describes how to appropriately introduce oneself to the community, explaining desirable behaviour (taking pictures, use of language, etc.), suitable clothing (many rural communities have a traditional and conservative way of dressing) and sensitive way of working with inhabitants and their community members (position of man, woman, adult and child). The second chapter describes a semi-structured interview with individual family members about their daily routine. It enables the practitioner[3] to better understand individual weekly schedules and daily life. More importantly, it helps to prepare and plan the activities to come (following the rest of the book).

The third chapter describes a play session with individual family members to unravel their hopes and dreams. As those most likely differ substantially from those of the professional; it helps to comprehend the family’s expectation from the project. The fourth chapter describes the general inventory (floor plans, sections, facades, etc.) of the existing house(s) and compound of the family. In the fifth chapter, individual family members are observed through one day of the week. In this way, the professional gets insight into where, how and with whom certain activities in and around the house are taking place.

The sixth chapter provides with an in-depth contextual analysis describing the inventory of housing typology in the area, basic geology and infrastructure. This analysis helps the professional to map available building materials and building methodologies. The seventh chapter helps the professional to prepare a structured interview to research the inhabitant’s capacities. This capacity analysis (together with the chapters on inhabitant involvement and knowledge transfer) is one of the most crucial methods developed within the authors’ PhD research. The method instructs how to evaluate available skills, finance, materials, tools and help from community members. Using the outcomes, the professional is able to develop three alternative housing solutions within the eighth chapter, which are based on the inhabitants’ existing housing situation, daily use, dreams, preferences and their capacities. After discussing the most suitable solutions with the family, the chapter results in a final design.

With many resources (materials, tool and labour) provided by the community, the ninth chapter helps to plan the building activities and include the identified materials, tools and labour. The last chapter of the support helps the professional to evaluate inhabitant skill levels per building activity, plan teaching/training activities and evaluate effective knowledge transfer to the inhabitants.

4. Quasi-experiment framework

Testing the support is an empirical investigation in how it influences the decision-making of both professional and inhabitant in improving the inhabitants’ self-reliance within their built environment. A complex set of variables (professional, inhabitants and context) and limited resources do not allow a random assignment (Keppel and Wickens, 2004). Here, the differences in financial capacities or family size could potentially make the research outcomes bias. To limit the variables in testing the developed support, the experiment is framed within a quasi-experiment (Bailey, 2008).

The experiments’ population size had to be restrained to the available resources, but also had to be large enough to prove a potential impact of the support and the potential societal benefit (Scher et al., 2015). For this reason, the total population size was set on four groups, consisting of one control group. Case group A has a positive impact and group B+C have a negative impact, there is sufficient evidence that the support does not have the desired effect. Without the third group, the results might become inconclusive (group A positive and group B negative). Cases group A+B have a positive impact and group C has a negative impact, however minimal, there is sufficient evidence for a positive trend. The control group D does not only help comparing outcomes with the groups that use the support, but also provides valuable information on the problems of the entire studied population (within Famia, community investigated on Mt Elgon).

The size of the individual teams was set by the support: the team has changing roles in which one team member is active as actor while the other observes and evaluated his team member. The team’s sampling was mainly based on financial constraints and convenience (Jager et al., 2017). It did not seem possible to find architects willing to pay for their own expenses while participating in the experiment. Covering expenses of the eight professionals was not possible and therefore an alternative had to be sought. Every year many students are involved in development aid; however, the added value for the supported communities can be questioned (Holdsworth and Quinn, 2010). Building engineering students were more than willing to participate in the experiment and pay for their expenses. They also contribute to one of the largest global groups of volunteers. However, they could not work according to the support without supervision. Therefore, the support has additional chapters for students and includes a working methodology combining one professional and one student.

The architects applied via a call that was set out in November 2016. The call was randomized and resulted in four architects originating from different countries: the Netherlands, Greece, Kenya and Kosovo. The architect selection procedure had two criteria: applicants need to hold an MSc in Architecture and need to be available during the entire experiment period. Selection of the architects was based on a heterogeneous convenience sampling (Jager et al., 2017), every applicant meeting the criteria was automatically selected.

The students applied via an internship call published on the virtual network of the Avans University of Applied Sciences and posters spread over both locations of the faculty. The student selecting procedure had three criteria: students needed to be third-year building engineer students and have finished the entire first-year curriculum. Selection of the students was based on a homogeneous convenience sampling (Jager et al., 2017), every applicant meeting the criteria was automatically selected.

After completing the selection for the experiment, the team composition was randomly sampled (Creswell, 2013). One jar consisted of small notes with the names of the architect and another jar with those of the students. The author was blindfolded and picked one note from each jar. In four rounds, the teams were randomly picked.

Due to the prior case studies and surveys, the quasi-experiment was situated on Mt Elgon. The survey evaluated the housing situation of 200 households within four different communities. Two selection criteria guided the choice of the community: inhabitants had to own the land they lived on and their current houses had to be built by locals (without external help). Results showed that in one community (red marker), the government owned the land. In another community (orange marker), an inhabitant collective owned the land and was heavily restrained in physical additions or changes to their housing situation. Most of the Chepchoina (green marker) inhabitants rented their house and did not own local land. The community members Famia (blue marker) owned their land and their houses were built by locals (or by inhabitants themselves); therefore, it was selected.

To find interested families in April 2017, a poster call was put on various locations in the Famia community (blue marker). People were asked to meet the following requirements:

• legally owning a piece of land;

• have family members available between August and January 2018 to help with construction;

• family members should be communicative in English;

• having and willing to contribute funds or materials for the building; and

• the plot should be within ten-minute motorbike ride from Andersen Medical Centre.

As families who applied had complex and differing capacity variables within a quasi-experiment setting, the assignment was nonrandomized. Therefore, four comparable families were sought. After two months, 14 families applied for the project. They were then visited by a local social worker who repeated the project requirements and participants’ responsibilities. One family (application no. 13) withdrew from the project, as they expected to receive a fully funded house to be built by the organisation. Multiple criteria were set for the families registering for the project; however, these criteria were not influencing the final selection. The criteria were: distance habitation teams, available budget, condition of the existing house, availability of materials, labour and time. The first selection round used a score system (0–10) for each criterion, based on the individual scores, seven families with the highest score were chosen.

While evaluating the selected applications, it appeared that some of the families already had a half built house, had an unclear family situation or had too few family members. These issues were so fundamental in realizing a new housing or they made too much difference in between the families that a new selection procedure was made. Again including all families and setting different criteria, prioritised by if the: family has already begun building a new house, availability of the family members, level of English, size of the plot, having children and budget. These criteria made sure that the family had the land to build a house on, basic financial means for small parts (tools, materials or labour) and a minimum level of English for basic communication between the team and the community members. Out of the 13 applications, 4 families were selected which had the most comparable scores. It must be stated that although the families are similar there are still substantial differences and, therefore, the experiment follows a non-equivalent group design (Kong et al., 2016; Moenaert and Caeldries, 1996; Wener, 1989).

The criteria of budget, amount of children and size of the plot have a considerable effect on the research outcomes (Figure 2). They defined how much financial means the family had, determining their expectations for the type of materials, size of the house and used building method for their new housing.

Assigning the families to the teams was also random sampling (Creswell, 2013). One jar consisted of small notes with the numbers of the teams and another jar with those of the families. The author was blindfolded and picked one note from each jar. In four rounds, the four families were linked to their team.

According to the code of ethics (Scientific Integrity Committee, 2012) of the Delft University of Technology, a separate research application was written together with partner institute Jomo Kenyatta University of Agriculture and Technology. The quasi-experiment was evaluated and approved by both institutes and later on by the National Commission for Science, Technology and Innovation (NACOSTI) in Kenya. The next section will describe the practicalities of executing the quasi-experiment on Mt Elgon.

5. Executing quasi-experiments

Executing a field experiment in a vulnerable context required many additional conditions to protect the family, community, team members and the quality of the gathered data. This section describes the following elements: governmental consent, community consent, family consent; media/financial/cancellation and team member consent/housing/office space/context introduction/nondisclosure agreement and communication.

According to the NACOSTI (2017) research permit legislation, the County Commissioner and Education Department were needed to approve the experiment. Therefore, upon their arrival both offices based in Kitale were visited to get the necessary official approval. The most important local level of approval came from the Areal Chiefs (Transnzoia), Community Board and Village Elder (Chepchoina). At this meeting the author was asked to explain the purpose of the experiment, the content of the support, potential participation of community members and the overall conditions to the research. The meeting was closed with the approval for the experiment to take place in Famia and a short welcoming ceremony for the research team to the community. Over the course of the project, three community meetings for were held in Famia. In these meetings inhabitants could ask any questions or address any remarks they had, concerning the experiment. During these meetings, there was always a local social worker and village elder present to guide the meeting.

Offering vulnerable families help raises many expectations and potential problems. To protect the family from making any decisions solely based on the external help, one of the most important processes in the quasi-experiment was the inhabitant informed consent (Mohler et al., 2010). During the first week, the author, assistant Researcher (Beata Duda) and Community Worker (Geoffrey Ngeywo) visited all families to informally introduce themselves and to hand out the consent form to the family (see Appendix 1). It included the following information: identification of the researcher, sponsoring institution, purpose of the study, identification of the benefits for participating, level and type of participant involvement, notation of risks to the participant, guarantee of confidentiality to the participant, assurance that the participant can withdraw at any time and provided the details of persons to contact if questions arise (Sarantakos, 2005). Moreover, the consent had a threefold varied description of the conditions to consolidate the inhabitants’ understanding of the conditions and making sure they could comprehend what they were signing. Together (researchers, family members and social worker) they read the entire document and then family members were able to address any questions, remarks or translations they might have had. The same procedure was followed for the audio/video consent (Appendix 2) to explain the procedures for recording and sharing data. The families were then left with the contract and audio/video consent for a couple of days to discuss them. When they were ready they contacted the author via the provided details and the zero measurement[4], an in-depth interviews (Muskat et al., 2012) were planned. In the next section the impact measurement is further explained.

After notifying participants about the acceptance of the experiments, multiple (Skype) meetings were held to go through all the conditions of the experiment. After two meetings, all participating team members were asked to read the contract and consent form. Afterwards they had the opportunity to ask questions or give suggestions to the contract. Subsequently, some of the projects conditions were changed in favour of the participants (originally all additional costs (visa, insurance, etc.) were to be financed by the participants themselves). The experiment described individual accommodation for every team, to reduce the risk of sharing information between each other. However, the group as a whole found the costs too high and separate accommodation as socially undesirable. Therefore, additional costs were covered by the author and a nondisclosure agreement was set up in which the teams agreed not to disclose any information between each other or to any third parties.

The teams were allowed to share everyday issues via social media and with each other; however, anything directly related to the experiment (research, design, decision-making, etc.) was prohibited. Due to the tremendous amount of preparation time, all teams required an office space through the first three months of the project. Although in the first weeks, there were some logistic issues every team got their own working space. Due to the high altitude and sensitive social/cultural context, the first week of the project was planned for context adjustment. Language courses (Swahili), desirable behaviour and dos and don’ts were addressed. Being a part of an impoverished community means that the teams had to be very conscious in what they say and do while being in the community (Liamputtong, 2006). During multiple sessions, the group discussed how to behave in a given scenario (community members asking for money, smoking and alcohol consumption in the community, etc.). At the end of the paper, an overview of considerations is given that occurred during the experiment and could help the effectiveness of similar future endeavours.

6. Impact measurement and implications of outcomes

To measure the outcomes of the tested support in a quasi-experiment setting, the framework targets the families involved in the research, as the overall aim being to improve inhabitant’s self-reliance towards their built environment. The most direct outcome of the experiment could be evaluated two to five years after its completion. Here, the factual maintenance, extension or reproduction of the house could be physically observed and easily made comprehensible (survey, interview or observation). However, the timeframe of the overall research is limited (PhD timeframe) and, therefore, a measurement directly after the project was necessary. To make the impact visible, a baseline measurement (Rubin and Babbie, 2009) was set up before the teams started to test the support. The measurable variables identified in the RC and DS1 phase intended to expose multiple issues concerning the inhabitants’ self-reliance in relation to the existing and desired housing. It included the following barriers to self-reliant housing:

• Housing size: many families prefer an expensive way of building often resulting in smaller housing, which cannot house the entire family and increases the inhabitant’s reliance on renting housing.

• Rent and landownership: In relation to variable 1, many families are forced to rent a house and land (Chepchoina) as they cannot afford to build an improved house, which increases the inhabitant’s financial reliance; no income=no habitation.

• Availability, locality and costs-building materials: traditional houses were built from free/cheap materials that were locally available, desired materials are expensive and not locally available. Building with them increases the inhabitant’s financial reliance.

• Self-build and hired labour: family and community traditionally build their houses (without financial compensation) themselves. As the desired materials are more complicated to work with, families and communities need to hire skilled labour, which increases the inhabitant’s financial reliance.

• Building knowledge: if the inhabitants do not have the required building knowledge, it increases their reliance to skilled labour.

• Maintenance and permanence: an often-heard complaint is the level of maintenance; the perception on modern housing[5] is that it does not require maintenance. However, it does, which increases the inhabitant’s reliance to skilled labour.

• Help and alternative solutions: the traditional building evolved over centuries while modernity has shown inhabitants a giant leap in housing quality and durability. As a result, almost the entire community desires a comparable style of “modern” housing. However, their financial capacities do not meet the needed requirements. This variable identifies the need of “external help” in finding alternative solutions according to the inhabitant’s capacities.

• Capacity acceptance: when looking at the financial means, inhabitants still prefer a housing solution that does not suit their capacities. An important additional variable pinpoints that if the inhabitant gained a better understanding of their actual capacities and related building solutions (more realistic desires towards their housing), then they are more likely to accept “cheaper” alternatives.

To better understand inhabitant’s motives concerning the variables, a structured in-depth interview was formulated (Appendix 3). This form for conducting interviews ensured that all measurable variables data are collected (Creswell, 2013) and in depth to allow the inhabitant to provide with additional and detailed motives (Guion et al., 2001). In the baseline interviews, inhabitants received questions about their current and desired habitation (Appendix 4). In the post-experiment (impact) interview, inhabitants got the same questions about current (new) and past habitation (Appendix 4). In both interview cycles, a social worker and the observer were present. Moreover, the inhabitants in both cycles received separate consent forms stating privacy issues and how the collected data would be used. The full outcomes of both the baseline and impact interviews are extensive and will therefore be described in detail in a consecutive paper by the author.

7. Conclusions

DRM proved to be a suitable model to evaluate existing problem in situ, analyse existing approaches, articulate a plausible support and successfully test the support in a quasi-experiment setup executed on Mt Elgon. The described procedures enabled the identification of families needing help in their housing and provided a setting in which they could safely participate in articulating improved housing. The procedures also successfully identified the participating practitioners and random sampling provided with an objective composition of the teams. Due to the extensive ethical approvals and setup of the experiment, there was a high level of awareness and involvement of both the local communities and government. Therefore, this paper provides valuable information under which circumstances design support can be tested and its impact in a vulnerable rural context could be evaluated. Providing researchers working in the field of empirical research with a workable setup to test applied support in situ, hopefully empowering practitioners intending to help rural inhabitants to sustain their self-reliance towards their housing preliminary outcomes confirm that there is a positive impact on the inhabitant’s self-reliance. However, due to the complexity of variables, the extensive data collection and transcripts, the outcomes of both interview cycles will be elaborated in a consecutive paper. This paper successfully described a quasi-experiment setup to test the design support within a DRM framework. The setup is specifically developed for the context of Mt Elgon; however, applying the setup in different contexts imposes substantial changes. Below, the main implications for application in comparable contexts are given:

• Vulnerable inhabitant’s social, cultural and financial capacities are context specific and might differ strongly with other communities.

• The described process procedures are specific for Kenya and might require different steps in properly engaging the government and community.

• Although nondisclosure agreements were signed, participants struggled not to breach contract. The experiment setting should ensure the participants’ discretion in future endeavours.

• Conducting this type of quasi-experiment over a period of five months requires a strong social group (moral support), which contradicts the conditions for discretion. The combination of participants into teams was very positive. However, this was primarily due to agreeable grouping and may have led to different outcomes; if the participants were grouped differently, the outcomes of the experiment could have been altered substantially. It is advised to use a personality assessment in pairing participants into teams (preferably under expert supervision).

Figures

Figure 1

DRM framework

Figure 2

Criteria and relative score per family: 0 failed criterion, 0.5 only partially met criterion and 1 fully meeting criterion

Notes

1. Capacities: all resources, knowledge and skills inhabitants have.

2. The support: the methodology developed within the author’s PhD research which entails: a book and multiple digital files (to be published after consecutive experiments and adjustments within the author’s postdoctoral research).

3. The practitioner: students and professionals working in the built environment.

4. Baseline measurement: measurement of identified variables at the beginning of the study, which is used to compare to later measurement for impact evaluation.

5. Modern housing: inhabitant perspective on improved housing in comparison to vernacular housing tradition.

Appendix 1

Appendix 2

Appendix 3

Appendix 4