Among the gold standards in human resource development (HRD) research are studies that test theoretically developed hypotheses and use experimental designs. A somewhat…
Among the gold standards in human resource development (HRD) research are studies that test theoretically developed hypotheses and use experimental designs. A somewhat typical experimental design would involve collecting pretest and posttest data on individuals assigned to a control or experimental group. Data from such a design that considered if training made a difference in knowledge, skills or attitudes, for example, could help advance practice. Using simulated datasets, situated in the example of a scenario-planning intervention, this paper aims to show that choosing a data analysis path that does not consider the associated assumptions can misrepresent findings and resulting conclusions. A review of HRD articles in a select set of journals indicated that some researchers reporting on pretest-posttest designs with two groups were not reporting associated statistical assumptions and reported results from repeated-measures analysis of variance that are considered of minimal utility.
Using heuristic datasets, situated in the example of a scenario-planning intervention, this paper will show that choosing a data analysis path that does not consider the associated assumptions can misrepresent findings and resulting conclusions. Journals in the HRD field that conducted pretest-posttest control group designs were coded.
The authors' illustrations provide evidence for the importance of testing assumptions and the need for researchers to consider alternate analyses when assumptions fail, particularly the homogeneity of regression slopes assumption.
This paper provides guidance to researchers faced with analyzing data from a pretest-posttest control group experimental design, so that they may select the most parsimonious solution that honors the ecological validity of the data.
Although pretesting is an essential part of the questionnaire design process, the range of methodological work on pretesting issues is limited. The present paper…
Although pretesting is an essential part of the questionnaire design process, the range of methodological work on pretesting issues is limited. The present paper concentrates on the effect of the pretest survey method on error detection by contrasting respondents who are interviewed personally with those who receive an impersonal survey method. The interaction between survey method and respondent knowledge of the questionnaire topic is also considered. The findings show that the pretest method does have an effect on the error detection rate of respondents; however, the hypothesised interaction between method and knowledge was not unequivocally supported. The detailed results illustrate which error types are affected by the method used during pretesting. Implications for future research are considered.
This article investigates the robustness of impulse response estimators to near unit roots and near cointegration in vector autoregressive (VAR) models. We compare…
This article investigates the robustness of impulse response estimators to near unit roots and near cointegration in vector autoregressive (VAR) models. We compare estimators based on VAR specifications determined by pretests for unit roots and cointegration as well as unrestricted VAR specifications in levels. Our main finding is that the impulse response estimators obtained from the levels specification tend to be most robust when the magnitude of the roots is not known. The pretest specification works well only when the restrictions imposed by the model are satisfied. Its performance deteriorates even for small deviations from the exact unit root for one or more model variables. We illustrate the practical relevance of our results through simulation examples and an empirical application.
Relative to the randomized controlled trial (RCT), the basic regression discontinuity (RD) design suffers from lower statistical power and lesser ability to generalize…
Relative to the randomized controlled trial (RCT), the basic regression discontinuity (RD) design suffers from lower statistical power and lesser ability to generalize causal estimates away from the treatment eligibility cutoff. This chapter seeks to mitigate these limitations by adding an untreated outcome comparison function that is measured along all or most of the assignment variable. When added to the usual treated and untreated outcomes observed in the basic RD, a comparative RD (CRD) design results. One version of CRD adds a pretest measure of the study outcome (CRD-Pre); another adds posttest outcomes from a nonequivalent comparison group (CRD-CG). We describe how these designs can be used to identify unbiased causal effects away from the cutoff under the assumption that a common, stable functional form describes how untreated outcomes vary with the assignment variable, both in the basic RD and in the added outcomes data (pretests or a comparison group’s posttest). We then create the two CRD designs using data from the National Head Start Impact Study, a large-scale RCT. For both designs, we find that all untreated outcome functions are parallel, which lends support to CRD’s identifying assumptions. Our results also indicate that CRD-Pre and CRD-CG both yield impact estimates at the cutoff that have a similarly small bias as, but are more precise than, the basic RD’s impact estimates. In addition, both CRD designs produce estimates of impacts away from the cutoff that have relatively little bias compared to estimates of the same parameter from the RCT design. This common finding appears to be driven by two different mechanisms. In this instance of CRD-CG, potential untreated outcomes were likely independent of the assignment variable from the start. This was not the case with CRD-Pre. However, fitting a model using the observed pretests and untreated posttests to account for the initial dependence generated an accurate prediction of the missing counterfactual. The result was an unbiased causal estimate away from the cutoff, conditional on this successful prediction of the untreated outcomes of the treated.
In the spirit of White’s (1982) paper, this paper examines the consequences of model misspecification using a panel data regression model. Maximum likelihood, random and…
In the spirit of White’s (1982) paper, this paper examines the consequences of model misspecification using a panel data regression model. Maximum likelihood, random and fixed effects estimators are compared using Monte Carlo experiments under normality of the disturbances but with a possibly misspecified variance-covariance matrix. We show that the correct GLS (ML) procedure is always the best according to MSE performance, but the researcher does not have perfect foresight on the true form of the variance covariance matrix. In this case, we show that a pretest estimator is a viable alternative given that its performance is a close second to correct GLS (ML) whether the true specification is a two-way, a one-way error component model or a pooled regression model. Incorrect GLS, ML or fixed effects estimators may lead to a big loss in MSE.
The purpose of this paper is to fill in a gap in the research literature on the subject of evaluation of coexistence programs.
Three separate research studies on the effects of joint sports programs on attitudes of Arabs and Jews toward each other were conducted from September 2011 to June 2012. Pretests and posttests were administered to participants in three sports programs involving Arabs and Jews: The Friendship Games, Mifalot's.
While pretests suggest that a great deal of hatred and lack of trust exists among Palestinians, Jordanians, and Israelis, according to the posttest results, by simply playing sports together, feelings of hatred can be greatly reduced and feelings of trust can be enhanced.
The research limitations included the need to shorten the questionnaire in order to encourage the youth to complete it, and the lack of control groups for the Friendship Games and Peres Center studies. More research is needed on this topic, as well as qualitative research to gain more insight into attitude changes.
The most important practical implication of the research is that conflict mitigation through sports programs and activities should be expanded in order to reach more people. The positive attitude changes found as a result of participation in the programs indicates that these joint sports programs really can promote better relations and if they are expanded to reach more people, the effects will be greater.
The originality/value of this study is great, as there has been almost no prior research which actually measured the effects on attitudes of youth of participation in integrated sports programs with Arabs and Jews.
In this paper we use Monte Carlo sampling experiments to examine the properties of pretest estimators in the random parameters logit (RPL) model. The pretests are for the presence of random parameters. We study the Lagrange multiplier (LM), likelihood ratio (LR), and Wald tests, using conditional logit as the restricted model. The LM test is the fastest test to implement among these three test procedures since it only uses restricted, conditional logit, estimates. However, the LM-based pretest estimator has poor risk properties. The ratio of LM-based pretest estimator root mean squared error (RMSE) to the random parameters logit model estimator RMSE diverges from one with increases in the standard deviation of the parameter distribution. The LR and Wald tests exhibit properties of consistent tests, with the power approaching one as the specification error increases, so that the pretest estimator is consistent. We explore the power of these three tests for the random parameters by calculating the empirical percentile values, size, and rejection rates of the test statistics. We find the power of LR and Wald tests decreases with increases in the mean of the coefficient distribution. The LM test has the weakest power for presence of the random coefficient in the RPL model.
The accounting profession has recognized the need to increase pro bono (free) tax preparation services. Increased pro bono tax preparation services by accountants may…
The accounting profession has recognized the need to increase pro bono (free) tax preparation services. Increased pro bono tax preparation services by accountants may address a growing unmet need for free tax preparation services by low-income taxpayers in the United States. One way to foster commitment to free services in the profession is by equipping accounting students with the knowledge and skills necessary to effectively serve low-income taxpayers in preparing their returns. We examined whether accounting students who provided free tax preparation services to low-income taxpayers as part of a service-learning course would experience significant changes in volunteering attitudes and motivation to offer free representation of low-income taxpayers in the future. The service-learning course was tied to the federal Volunteer Income Tax Assistance (VITA) program. Results from traditional pretest/posttest and retrospective pretest/posttest found participants reported significant increases in volunteering attitudes, the role of accounting in addressing social issues, attitudes toward helping others, and motivation and competence to offer future free representation of low-income taxpayers. Also, most participants reported positive experiences in the VITA clinic and further developed skills important to the accounting profession. Qualitative data supported the quantitative results.
Presents a study that examined the effects of cross‐cultural instruction on the interpersonal job skills of students in secondary vocational programs. The population…
Presents a study that examined the effects of cross‐cultural instruction on the interpersonal job skills of students in secondary vocational programs. The population consisted of a treatment and a control group, with 65 students in each group. A pretest and posttest was administered. The experimental group received the intervention during a six‐week period. The dependent variable was generalizable interpersonal relations skills as measured by the generalizable interpersonal relations skills performance assessment. The independent variables included a cross‐cultural instructional intervention, gender, ethnicity, and school. The findings indicated that students receiving the cross‐cultural instructional intervention had significantly higher generalizable interpersonal relations skills achievement than students not receiving the intervention. Recommends that cross‐cultural instruction be integrated into vocational and industrial training curricula, including instructor preservice and inservice training. Also, trainers should closely link the benefits of cultural awareness to learners’ experiences at home and the work place; and to their personal and professional success.
To address a persistent crime problem in a large, high crime apartment complex occupied by Latino and Vietnamese immigrants in Garden Grove, California, the local police…
To address a persistent crime problem in a large, high crime apartment complex occupied by Latino and Vietnamese immigrants in Garden Grove, California, the local police department opened a storefront office. The purpose of this study was to examine the impact this community policing effort had on the residents of the complex in terms of reducing fear of crime and improving attitudes towards the police. A questionnaire was administered to the residents before opening the storefront office and at the conclusion of the project. Despite group differences, the results revealed that the intervention had a positive impact on both groups. Overall, Latinos reported less fear of crime after the intervention and improved perception of the police. The Vietnamese reported more fear of crime than Latinos (pre and post) but less fear of crime after the intervention. However, their attitudes towards the police, which were very positive on the pretest, did not improve on the post‐test.