Which story to tell: applying age, period, and cohort approaches to leadership assessment

Abstract

Purpose

We highlight three approaches for structuring data analysis to aid leadership educators and researchers in investigating differences between populations, considering the variable of age.

Design/methodology/approach

Utilizing real data, we exemplify the three approaches to illustrate how insights might be gained.

Findings

We offer illustrative empirical findings in this reflective essay to demonstrate the three approaches. Our empirical examples are real, but not designed to be the purpose of this essay.

Research limitations/implications

We provide three methodological approaches to analyzing leadership data that can assist leadership educators and researchers in determining an appropriate method for meaning-making with their data.

Originality/value

We seek to describe three different approaches to data analysis that are likely accessible and convenient as well as could lead to insight for leadership educators and researchers.

Citation

Seemiller, C. and Rosch, D.M. (2024), "Which story to tell: applying age, period, and cohort approaches to leadership assessment", Journal of Leadership Education, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/JOLE-05-2024-0070

“Many young people don’t vote.” This line is stripped from the headlines of countless news articles as intriguing clickbait or a call to action. Whether the claim is true or not, how does one make that assertion? Is this referring to young people today compared to young people 20 years ago? Young people compared to older people right now? Or, young people 20 years ago compared to older people today, insinuating a “growing into voting” phenomenon?

Depending on the way one looks at the data, different stories may emerge. Thus, knowing which question to ask is of paramount importance. While these nuances have helped guide researchers in looking uniquely at data to make determinations about specific generational groups (Pew Research Center, 2015), can doing so aid in better assessing leadership development?

Our goal in this essay is to describe methods for analyzing leadership data using three approaches – age, period, and cohort, and describe how these methods can be differentially applied. We do not imply that these individual methods are wholly absent from leadership assessment efforts, but there is scant support that the age, period, and cohort framework is utilized in the field of leadership studies (e.g. Parry, Mumford, Bower, & Watts, 2014). Further, given its widespread use in other fields, we assert that leadership educators in both student affairs and in academic programs could benefit from a comprehensive understanding of the three approaches and how to apply them to program-level assessment efforts.

Age, period, and cohort approaches

According to the Pew Research Center, “‘age’ denotes a person’s stage in the life cycle, ‘period’ refers to when the data was collected, and ‘cohort’ refers to a group of people who were born within the same time period” (2015, para. 6).

Age approach

The “age” approach, sometimes referred to as lifecycle, focuses on change over time. Fundamentally embedded within this approach is the idea that people change in relatively expected ways as they mature based on the similarities that make them human (Erikson, 1993). A typical assessment structure with the age approach is a straightforward longitudinal, repeated-measures analysis where a participant is invited to respond to the same measures on numerous occasions over time, and where a trajectory of scores can be assessed and then analyzed. While connecting responses from specific individuals to their previous ones is a common procedure for employing the “age approach,” it is not necessary as long as group scores are collated and compared over time.

An “age” approach might be employed to compare growth of a specific group of participants from measures taken after the first semester of a leadership program to scores after the second semester, and so on, allowing leadership educators to better anticipate developmental changes that occur with students over the span of the program. This approach can be applied in any learning context as long as the data collected takes place where stable curricula also exist (i.e. when the assessed program does not change drastically year to year).

Period approach

A second means for structuring an analysis is labeled the “period” approach, where data from differently-aged groups of people are collected during the same time period and used for comparison. Employing a period approach, leadership educators and researchers might compare leadership competency scores of 18-year-olds with those of 21-year-olds after participating together in a semester-long certificate program. In this case, each student experienced the same curriculum at the same time, however, the differing variable is their age.

After data is analyzed, it could be determined that the 21-year-olds have far greater gains after participation than the 18-year-olds, thus supporting a shift to offer this particular certificate to juniors and seniors only. Or, the reverse could be the case, indicating a need to target the program to first-year students.

Cohort approach

The last method for analyzing data is the “cohort” approach. Cohorts are groups of people who have a shared understanding of the world around them, particularly in response to experiencing societal events around the same age (Pew Research Center, 2015). Thus, cohorts develop defining characteristics shaped by these experiences (Mannheim, 1952); these characteristics then make up a generational consciousness or “we-identity” (Connolly, 2019).

While generational terms like Baby Boomer, Generation X, Millennial, and Generation Z are often used to describe groups of individuals born within a specific time frame, cohorts need not be limited to societal labels and span decades. For example, within a campus-based leadership program, a cohort could be defined as “18-year-old students in 2021.” It might be important to investigate how this class of students might have experienced the first semester of the leadership program differently than those who were 18 five years prior in 2016. The key to using a cohort approach is that data from groups of the same age or stage are collected during different time periods and used for comparison. This can help determine any differences between the cohorts.

A cohort approach can be particularly helpful in investigating how a campus population’s attributes might have shifted over time. This type of analysis might be particularly relevant now, in the wake of the pandemic, where, for example, first-year students pre-COVID might have had a qualitatively different experience than those post-COVID, even though both groups engaged in the same leadership program at the same age.

Utility of age-period-cohort approaches

Age-period-cohort methods have been proven to serve as a robust way of measuring time-varying data (Columbia University, n.d.) and have been used for decades (e.g. Blanchard, Bunker, & Wachs, 1977; Hobcraft, Menken, & Preston, 1982) across a variety of disciplines, including health (Bell & Jones, 2015), sociology and demography (Bozick, 2021), and psychology (Keyes et al., 2014).

Looking at data through one of these lenses can produce results that are vastly different than if the data were analyzed through another, impacting which story is essentially told. While utilizing one or two of these methods in isolation can be useful (Fitzenberger, Mena, Nimczik, & Sunde, 2021), some scholars have developed robust methods for accounting for the interrelationship between them (e.g. Columbia University, n.d.; Yang & Land, 2013; Bell, 2020). Thus, if there is data that can be analyzed using all the methods, statistical models are available to explore this complexity.

Illustrative example of the approaches

Now that we have described these three approaches, we can illustrate examples with real data from the Student Leadership Competencies Inventory (Seemiller, 2015) collected from nearly 10,000 participants. In the Inventory, three survey items are used to create an average score for each of the 60 competencies measured, and participants are given a report after completing each assessment. The Inventory is free and has been psychometrically validated, with a publication detailing those findings (Rosch & Seemiller, 2018). For example, each item has been shown to possess strong internal reliability (most over 0.80), while confirmatory factor analysis statistics also indicate a strong underlying factor structure within each sub-scale (CFI statistics over 0.90 and RMSEA findings between 0.05 and 0.07).

Using this dataset, we compared mean proficiency scores of the competency, “responding to ambiguity,” which includes the following three items:

• (1)

  “I can effectively function in situations of uncertainty.”

• (2)

  “Even if I don’t have all the information I would like, I can still move forward in making a decision that needs to be made.”

• (3)

  “I am comfortable working on tasks or projects that have little direction or explanation.”

Each item included a 7-point Likert-scale response set, where 1 = Strongly Disagree and 7 = Strongly Agree. Overall proficiency scores for “responding to ambiguity” are listed in Table 1.

Age approach

How might a student’s perception of their ability to respond to ambiguity change as they get older? With this approach, we can investigate trajectories of development, ascertaining important times when growth might be occurring. To do this, we examined scores on a left-to-right diagonal, or 19–21-year-olds in 2016 (4.56), 22–24-year-olds in 2019 (5.22), and 25–27-year-olds in 2022 (5.13). In this case, the age approach was used with different groupings of students for each subsequent time block, such that a 20-year-old student who completed the Inventory in 2016 was not the same 23-year-old who took it in 2019. However, this method can be employed with longitudinal data that is collected from the same students at multiple points in time.

In this example, scores increased from the time that many students were in college to their post-graduation years, where they seemed to plateau. One possible explanation for the growth trajectory may be that as individuals accumulate more life experience, they become faced with more ambiguous opportunities and more tested strategies for dealing with them. However, with the COVID-19 pandemic occurring between the last two data points, it is unclear whether that trajectory would have continued in 2022, rather than remained relatively static, had circumstances been different.

Period approach

What maturation factors might affect a student’s perception of their level of proficiency in terms of responding to ambiguity? Using this approach allows us to directly compare participants of different ages at a certain snapshot in time, investigating the degree to which age (and maturity) might matter. In this case, we compared scores within the horizontal rows, or 19–21-year-olds, 22–24-year-olds, and 25–27-year-olds, measured in each of the time blocks (2016, 2019, and 2022).

Unsurprisingly, scores seemed to rise a bit overall as participant ages increased. It might seem logical to presume that older humans might be more skilled at responding to ambiguity given their longer life experience, and therefore practice in responding in such an environment. However, it is important to note that scores dipped between 2019 and 2022, which could again be related to experiences during the COVID pandemic.

Cohort approach

Why did students who were in college in 2019 indicate having higher rates of proficiency regarding responding to ambiguity compared to those in college in 2016? This approach allows us to investigate the degree to which “time in history” matters, taking into consideration any significant events during those eras that might influence the scores. To do this, we noted changes in scores in the “Ages 19–21” column at 4.56 in 2016 to 5.20 in 2019 and 5.17 in 2022. Is it possible that the increasing complexity of navigating an ambiguous world in 2019 was related to the uncertainty of an upcoming election or the proliferation of the social media app, TikTok, which features influencers questioning the status quo? Further, perhaps the slight differences between the 2019 and 2022 scores, specifically, might have been impacted by COVID.

While this essay’s focus is not to offer an interpretation of our illustrative example, we highlight that utilizing these methodologies can help point researchers in a direction for further exploration and help delineate which story they want to tell.

Using the age-period-cohort methods in leadership research

Included are the steps for deploying age-period-cohort approaches in leadership assessment.

• (1)

  Decide whether to use age, period, and/or cohort approaches.

  • •

    Age: Tracks a group of people as they age (e.g. Follow first-year students through their senior year to measure their leadership development with a fixed variable, like ethical decision-making).

  • •

    Period: Measures a phenomenon at a specific point in time with people across all age groups (e.g. Comparing learning gains with different age groups at the end of a leadership course).

  • •

    Cohort: Compares data with the same-aged people over time (e.g. Average volunteer hours of first-year students every year for four years)

• (2)

  Determine the age grouping(s) (e.g. Specific age, age span, or year in school).

• (3)

  Determine the time block(s) (e.g. Beginning and end of a school year, each semester, after each workshop, or calendar year).

• (4)

  Select the type of data to compare (e.g. means, percentages, or hours towards completion).

Again, while deploying only one approach can garner insightful information, it might be useful to analyze the data using additional methods, if possible. Doing so offers the opportunity to look at the data from different vantage points and determine which story to tell.

Conclusion

In this essay, we aimed to describe three different approaches for analyzing data to ascertain differences between and across groups of students engaged in leadership programs. To be clear, we do not suggest that these assessment structures are not existent within the field. In fact, leadership assessment officers might already be employing one or two of the approaches in their own work without realizing their connections to the larger age-period-cohort framework. And those who are not might find value in learning a methodology that makes better and more intentional use of the time-varying data they might already be collecting or offer them a blueprint for future data collection and analysis.

Table 1

“Responding to ambiguity” proficiency mean scores

	Ages 19–21	Ages 22–24	Ages 25–27
2016	4.56 (n = 145)	4.79 (n = 92)	4.83 (n = 32)
2019	5.20 (n = 302)	5.22 (n = 213)	5.66 (n = 54)
2022	5.17 (n = 347)	5.31 (n = 316)	5.13 (n = 94)

Source(s): Table by authors