Academic education

, 33–34

Academic expertise

, 33–37

Academic freedom

, 123

Academically Adrift

, 67

Accountability

, 60, 108, 110–111

issue

, 7

processes in higher education

, 19

ACT

, 62

Affective, behavioural and cognitive learning gains (ABC learning gains)

, 2

Affective measures

, 105

Affective-motivational facets

, 34

Ambiguity of learning outcomes

, 10

American Association of State Colleges and Universities (AASCU)

, 60

Analysis of covariance (ANCOVA)

, 26–27

Analysis of variance (ANOVA)

, 26–27

Assessment of Higher Education Learning Outcomes project (AHELO project)

, 102

Assessment of Learning Outcomes in Higher Education (AHELO)

, 38

Association for Advancement of Sustainability in Higher Education (AASHE)

, 124

Association for American Colleges and Universities (AAC&U)

, 61

Association of German Chambers of Industry and Commerce (DIHK)

, 37

Association of Public and Land-grant Universities (APLU)

, 60

Australian higher education sector

, 132–133

challenges facing

, 133–136

Australian institutions

, 131–132, 146

Australian Qualifications Framework (AQF)

, 38–39

Australian universities

, 132, 135

collaborative approach to promoting excellence and innovation

, 140–143

enabling educators

, 143–146

government role in promoting and influencing learning and teaching quality

, 136–140

insights from equity initiatives

, 146–148

transition pedagogy for learning gain

, 146–148

Average normalised gain

, 25

Behavioural change

, 121–122

Behavioural measures

, 105

Big data

, 104

Bologna Declaration

, 90

Capacity building

, 84–85

Capstone Project

, 161–162

Chemical Concepts Inventory (CCI)

, 20–21

Cognition

, 34

Cognitive abilities, measures of

, 105

College Scorecard

, 61

Collegiate Assessment for Academic Proficiency (CAAP)

, 62

Collegiate Learning Assessment (CLA)

, 38–39, 62, 102

Collegiate Learning Assessment Plus (CLA+)

, 162

Commission on Future of Higher Education

, 60

Commonwealth Grants Scheme (CGS)

, 132, 136–137

Commonwealth Supported Places (CSPs)

, 132

Competence

, 120, 122–123

development

, 49

portfolio

, 34–35

Competencies

, 120, 122–123, 127

Competency assessment experts

, 95–96

Computer-based assessments

, 49

Constructs

, 118

Continuing professional development (CPD)

, 85

Control groups

, 83

Cooperative Institutional Research Program

, 94

Council for Aid to Education (CAE)

, 62, 66–67, 162

Course Experience Questionnaire (CEQ)

, 139

Course learning outcomes and scoring

, 161–162

Critical thinkers

, 128

Cross-sectional designs

, 64–65

Curriculum design

, 141

Data collection

, 157–158

Deep knowledge

, 159–160

Domain-specific expertise. See Academic expertise

Education

, 119, 154

Education for sustainable development (ESD)

, 119–120

Educational research

, 123

Educational Testing Service (ETS)

, 38–39, 62, 70

Effectiveness

, 106

ELMaWi

, 50

Employability

, 88–89, 91–93

measures

, 105

performance-based tests

, 93, 95–96

self-reports

, 93–95

skills

, 93

Employer Satisfaction Survey (ESS)

, 139

Employment

, 91–92

English higher education sector

, 103

English higher education system

, 109, 112

Equity initiatives

, 146–148

Equivalent fulltime student load (EFTSL)

, 134

ERIC database

, 2

ETS Proficiency Profile (EPP)

, 7, 62, 65–66

Europe of knowledge

, 89

European Commission

, 90, 127

Evaluation of learning gain

, 103–107

Excellence

, 140–143

Excellence in Assessment (EIA)

, 61, 71–72

External validation from employers

, 163–164

Federal governments in Australia

, 132

Feedback, MSKGI

, 157–159

Financial incentives

, 83

Force and Motion Concept Evaluation test (FMCE)

, 20–21

Full information maximum likelihood (FIML)

, 68–69

German Council of Science and Humanities

, 108–109

German Federal Ministry of Education and Research (BMBF)

, 38–39, 108–109

German higher education

first working phase (2011–2015)

, 39

modeling and measuring competences

, 38–40

second working phase (2016–2020)

, 40

Google Scholar

, 2, 4–5

Grades

, 102

Graduate Outcome Survey (GOS)

, 139

Graduates

, 133, 135, 139

Habits of Mind and Foundational Concepts (HCs)

, 156, 158, 161

Hawthorne effect

, 82–83

HEIghten Critical Thinking assessment

, 73

HEIghten Outcomes Assessment Suite

, 63

Hierarchical linear modeling (HLM). See Multilevel modeling (MLM)

Higher education

, 4, 17–18, 88

accountability processes in

, 19

analysis focus and research questions

, 35

challenges in measuring students’ learning outcomes

, 35–36

changing role of

, 89–91

directions for future research

, 52

institutional contexts and shortcomings in Europe and Germany

, 36–38

institutions

, 71

learning gain in

, 4–6

research

, 2–3

students’ competences and learning outcomes in

, 33–35

traditional role

, 88

transfer in higher educational practice

, 50–52

Higher education for sustainable development (HESD)

, 118, 120

future for

, 126–128

research demonstrating overall lack of progress

, 125

research demonstrating some progress

, 124–125

research on barriers

, 123–124

in twenty-first century

, 118–125

Higher Education Funding Council for England (HEFCE)

, 77, 102

HEFCE/OfS

, 78

programmes

, 79, 84

Higher Education Research and Development Survey

, 61

Higher Education Statistics Agency (HESA)

, 109

Hiring managers in Minerva

, 163–164

‘Humboldtian’ model

, 90

In-depth validation

, 40

Innovation

, 140–143

Innovative technology-based formats

, 49

Institutional contexts in Europe and Germany

, 36–38

Institutional effectiveness

, 71–72

Institutions

, 61–62, 85, 112

Integrated Learning Outcomes (ILOs)

, 164

of Minerva

, 165

Integrated Postsecondary Education Data System (IPEDS)

, 61

Internal structure

, 42

Keck Graduate Institute (KGI)

, 155

Knowledge

, 122

transfer

, 161

.Knowledge

, 159–160

KoKoHs II

constructs assessment

, 47

limitations and implications for future research

, 49–50

research areas

, 40–47

results and outcomes

, 48–49

types of test instruments and of application areas

, 48

.Lacks knowledge

, 159–160

Learner-centred curriculum design

, 133

Learning

collaborative approach to promoting excellence and innovation in

, 140–143

in context of HESD

, 121–122

government role in promoting and influencing

, 136–140

losses

, 118

Learning and Employability Gain Assessment Community Project (LEGACY)

, 77–78

building communities of understanding learning gain

, 85

capacity building

, 84–85

control groups

, 83

evaluation methodology

, 78–79

financial incentives

, 83

Hawthorne effect

, 82–83

measuring learning gain

, 81–84

mixed methodologies

, 82

project success criteria

, 79–80

qualitative approaches

, 82

quantitative approaches

, 81

standardised tests

, 81–82

student engagement

, 83–84

Learning and Teaching Repository

, 142

Learning Attitudes about Science Survey (CLASS)

, 21

Learning gain

, 2, 3, 17–18, 64, 77, 101, 112–113

application and practice

, 4–8

building communities of understanding

, 85

challenges for higher education sectors

, 110–112

challenges for universities

, 110

challenges to measuring

, 107–112

comparison between groups

, 26–27

computation of

, 23–26

cross-sectional vs. longitudinal designs

, 64–65

design challenges

, 109–110

development of measures

, 103–106

effectiveness

, 106

evaluation

, 103–107

intellectual and political challenges

, 108–109

issues and critique

, 8–11

LEGACY project and

, 81–84

measuring

, 19–23

metrics

, 102

origins and meaning

, 3–4

robustness

, 106

scalability

, 107

suitability

, 107

trajectories across 10 STEM Modules

, 24

transition pedagogy for

, 146–148

Learning outcome (LO). See also Student learning outcome (SLO)

, 6, 122

ambiguity of

, 10

measurement of Minerva

, 164

MSKGI

, 156–157

Life post undergraduate in Minerva

, 163–164

Longitudinal designs

, 64–65

Manifest

, 162

Mastery Rubric template

, 159

Maximum likelihood estimation

, 68–69

Mean imputation

, 68

Measurement instruments

, 39, 50

Methodological evaluations

, 40

Methodological skills

, 37

Mid-year economic and fiscal outlook (MYEFO)

, 137, 138

Minerva. See Minerva Schools at Keck Graduate Institute (MSKGI)

Minerva Forum™

, 155–156, 158

Minerva Schools at Keck Graduate Institute (MSKGI)

, 154–156, 154

Capstone Project

, 161–162

complement with external validation/assessments

, 162–163

data collection and feedback

, 157–158

external validation from employers

, 163–164

hiring managers

, 163–164

integrated learning outcomes

, 165

knowledge transfer

, 161

life post undergraduate

, 163–164

measurement and assessment

, 156–157

measurement of additional learning outcomes

, 164

NS111 implications of earth’s cycles

, 159–161

student assessments

, 159

Missing completely at random (MCAR)

, 68

Missing data on assessment tool

, 68–69

Mixed methodologies

, 82, 102

Mobility

, 70

Modeling and Measuring Competences in Higher Education (KoKoHs). See also KoKoHs II

, 34–35, 38–40, 50–51, 108–109

additional constructs assessed in 16 collaborative projects

, 42

research clusters with 16 collaborative projects

, 41

Mplus

, 68–69

Multilevel modeling (MLM)

, 66–67

Multiple imputation

, 68–69

National Association of Colleges and Employers (NACE)

, 73

National Center for Evaluation of Higher Education (CENEVAL)

, 38–39

National Institute for Learning Outcomes Assessment (NILOA)

, 61–62, 71–72

National Research Council (NRC)

, 34, 36

National Student Clearinghouse Research Center

, 70

National Student Survey (NSS)

, 19

National Survey of Student Engagement (NSSE)

, 62

Normalised change

, 25–26

Normalised gain

, 25

NS111 implications of earth’s cycles

, 159–161

NVivo

, 78–79

Objective measurements

, 8

Objective Structured Clinical Examinations (OSCE)

, 95, 122–123

Office for Learning and Teaching (OLT)

, 137–138, 140–141

OLT-funded projects

, 141

Office for Students (OfS)

, 103

Ordinary least squares regression (OLS regression)

, 65–66

Organization for Economic Co-operation and Development (OECD)

, 38, 89, 102

Overpopulation

, 118

Performance-based funding

, 108, 137–138

Performance-based tests

, 93, 95–96

Personal development

, 33–34, 37

Policy makers

, 88

Politics

, 108–109

“Portfolio of KoKoHs Assessments”

, 48–49

Post-tests

, 20, 22

Pre-tests

, 20, 22

Problem-solving tests

, 105

Professional, statutory and regulatory bodies (PSRBs)

, 112

Professional Standards Framework

, 120–121

Professional training

, 33–34, 37

.Profound knowledge

, 159–160

Programme for International Assessment of Adult Competences (PIACC)

, 38

Promotion of Excellence in Learning and Teaching in Higher Education (PELTHE)

, 140

Propensity score matching

, 64–65, 67

Qualitative approach

, 2, 82

Qualitative methods

, 102

Quality assurance

, 93, 102, 111–112

Quality enhancement

, 102

Quality Indicators for Learning and Teaching (QILT)

, 139

Quantitative approaches

, 81

Quasi-experimental learning gains research

, 20–21

Queen Mary University London (QMUL)

, 85

Reliability

, 106

Representativeness of learning gain

, 24

Research

on barriers

, 123–124

demonstrating overall lack of progress

, 125

demonstrating some progress

, 124–125

Residual gain

, 25

Rilevazione Continua delle Forze di Lavoro’ (RCFL)

, 111

Robustness

, 106

Rubrics

, 159

comprehensive

, 157

ILO

, 165

Sampling/recruiting students

, 70

Scalability

, 107

Scale

, 132, 135–136, 144–145

Science, technology, engineering and mathematics (STEM)

, 5–6

Scopus databases

, 2

Self-reported learning gains (SRLG)

, 9

Self-reports/self-reporting

, 93–95

surveys

, 102

sustainability performance

, 124

Shortcomings in Europe and Germany

, 36–38

Skills

, 122

Social network analysis

, 141–142

Socio-economic status (SES)

, 134

Spellings Commission

, 73

Standardised tests

, 81–82, 102

Standards for Educational and Psychological Testing

, 39

Student Achievement Measure (SAM)

, 61

Student Assessment of Learning Gains survey (SALG)

, 21

Student assessments

, 159

Student attrition

, 70

Student diversity

, 147–148

Student engagement

, 83–84, 101, 107

Student Experience Survey (SES)

, 139

Student learning

, 110–111

assessment tools for measuring value added

, 63–64

assessments tools for measurement

, 61–64

caution in making causal assumptions about

, 71

challenges in measuring

, 68–70

institutional effectiveness

, 71–72

preparing students for workforce

, 72–73

VSA recommended assessments for measuring value added

, 62

Student learning outcome (SLO). See also Learning outcome (LO)

, 7

challenges in measuring

, 35–36

in higher education

, 33–35

Student motivation on learning outcomes assessments

, 69–70

Student Opinion Scale (SOS)

, 69–70

Student representation

, 70

sampling/recruiting students

, 70

student attrition and mobility

, 70

Students’ competences

continuum of

, 47

in higher education

, 33–35

Suitability

, 107

Summative assessments

, 157

.Superficial knowledge

, 159–160

Sustainability Tracking, Assessment & Rating System) (STARS)

, 124

Sustainable development goals (SDGs)

, 119

Systematic enquiry

, 125

Systematic review

, 2

Teaching

collaborative approach to promoting excellence and innovation in

, 140–143

enhancement

, 110–111

excellence

, 101

government role in promoting and influencing

, 136–140

Teaching Excellence Framework (TEF)

, 18, 120–121

Tertiary Education Quality and Standards Agency (TEQSA)

, 136, 145

Transfer in higher educational practice

, 50–52

Transition pedagogy

, 147–148

for learning gain

, 146–148

Transparency

, 60

Transparency Framework

, 61, 71–72

True gain

, 25

Twenty-first century skills

, 34

United States of America (USA)

, 60, 89–90

Universities

, 136

enabling educators

, 143–146

teachers

, 120–121

University of California, Irvine (UCI)

, 73

University of Cambridge

, 85

University of Manchester

, 85

University of Warwick

, 82–83

Research Code of Practice

, 78–79

US Department for Education

, 36

US Secretary of Education Margaret Spellings

, 60

Usability

, 107

Valid Assessment of Learning in Undergraduate Education (VALUE)

, 63–64

Validity

, 106

Value added in higher education

assessment tools for measuring

, 63–64

cross-sectional vs. longitudinal designs

, 64–65

decreased focus on

, 71–72

holistic approach to assessment

, 72

measurement

, 64

MLM

, 66–67

OLS regression

, 65–66

propensity score matching

, 67

VSA recommended assessments for measuring

, 62

Voluntary System of Accountability (VSA)

, 60–61, 63–64, 71–72

recommended assessments for measuring value added

, 62

VSA Analytics

, 61

WASC Senior College and University Commission (WSCUC)

, 155

Web of Science database

, 2

Weighting

, 68–69

Workforce, preparing students for

, 72–73