Search results1 – 6 of 6
This chapter focuses on the state-of-the-art modeling approaches used in Intelligent Tutoring Systems (ITSs) and the frameworks for researching and operationalizing…
This chapter focuses on the state-of-the-art modeling approaches used in Intelligent Tutoring Systems (ITSs) and the frameworks for researching and operationalizing individual and group models of performance, knowledge, and interaction. We adapt several ITS methodologies to model team performance as well as individuals’ performance of the team members. We briefly describe the point processes proposed by von Davier and Halpin (2013), and we also introduce the Competency Architecture for Learning in teaMs (CALM) framework, an extension of the Generalized Intelligent Framework for Tutoring (GIFT) (Sottilare, Brawner, Goldberg, & Holden, 2012) to be used for team settings.
Increased interest in team dynamics has resulted in new methods for measuring teamwork over time. The primary purpose of this chapter is to provide a survey of recent…
Increased interest in team dynamics has resulted in new methods for measuring teamwork over time. The primary purpose of this chapter is to provide a survey of recent developments in teamwork/collaboration measurement in an educational context. Key topics include conceptual frameworks, large-scale assessments, and innovative measurement techniques.
A range of methods for collecting and analyzing teamwork data are discussed, and five frameworks for measuring collaborative problem solving (CPS) over time are compared. Frameworks from Programme for International Student Assessment (PISA), Assessment and Teaching of 21st Century Skills (ATC21S) project, Educational Testing Service (ETS), ACT, and von Davier and Halpin (2013) are discussed. Results of assessments developed from these frameworks are also considered.
New techniques for measuring team dynamics over time have great potential to improve education and work outcomes. Preliminary results of the assessments developed from these frameworks show that important advances in teamwork measurement have been enabled by innovative task designs, data-mining techniques, and novel applications of stochastic models.
This novel overview and comparison of interdisciplinary approaches will help to indicate where progress has been made and what challenges are ahead.
Graduate admission has become a critical process for quality assurance in tertiary education. Hitherto, most research has investigated the validity of admissions…
Graduate admission has become a critical process for quality assurance in tertiary education. Hitherto, most research has investigated the validity of admissions instruments. However, surprisingly little work has been conducted on the overall organization of admission, which often remains “informal, ad hoc, and lacking in continuity.” The purpose of this paper is to investigate how to systematically design an admissions process for effectiveness, fairness, and the ability to continually improve, and determine how to condense and analyze the massive amount of data available from student records to obtain high-value feedback for admissions decision making.
An admissions process was systematically designed based on results from process management theory. Tenets of decision theory were applied to the organization of decision making and validity theory was utilized for validating admissions instruments. Performance of the implemented process was evaluated via student records covering a seven-year period.
The authors have designed a four-phase admissions process that ensures high quality through screening, scoping, selection, and evaluation/feedback. The last phase introduces closed-loop control and facilitates stabilization and continual improvement. Additionally, the authors have established a three-stage decision-making hierarchy that promotes consistency and equal treatment in admissions. The evaluations of undergraduate achievements and GRE® General Test scores indicate that both are valid admissions instruments in the European context. Finally, the evaluation of the implemented process provides evidence that decision making has effectively improved over the years and has become more consistent.
The systematic design of the admissions process presented generalizes well and is a significant contribution to the organization of decentralized graduate admission.
This policy brief discusses validity and fairness issues that could arise when test-based information is used for making “high stakes” decisions at an individual level…
This policy brief discusses validity and fairness issues that could arise when test-based information is used for making “high stakes” decisions at an individual level, such as, for the certification of teachers or other professionals, or when admitting students into higher education programs and colleges, or for making immigration-related decisions for prospective immigrants. To assist test developers, affiliated researchers and test users enhance levels of validity and fairness with these particular types of test score interpretations and uses, this policy brief summarizes an “argument-based approach” to validation given by Kane.
This policy brief is based on a synthesis of conference proceedings and review of selected pieces of extant literature. To that synthesis, the authors add practitioner-friendly examples with their own analysis of key issues. They conclude by offering recommendations for test developers and test users.
The authors conclude that validity is a complex and evolving construct, especially when considering issues of fairness in individual testing contexts. Kane's argument-based approach offers an accessible framework through which test makers can accumulate evidence to evaluate inferences and arguments related to decisions to be made with test scores. Perspectives of test makers, researchers, test takers and decision-makers must all be incorporated into constructing coherent “validity arguments” to guide the test development and validation processes.
Standardized test use for individual-level decisions is gradually spreading to various regions of the world, but understandings of validity are still uneven among key stakeholders of such testing programs. By translating complex information on test validation, validity and fairness issues with all concerned stakeholders in mind, this policy brief attempts to address the communication gaps noted to exist among these groups by Kane.