Analysis of performance on a modified Wisconsin Card Sorting Test for the military

Purpose 
 
 
 
 
Current Army doctrine stresses a need for military leaders to have the capability to make flexible and adaptive decisions based on a future unknown environment, location and enemy. To assess a military decision maker’s ability in this context, this paper aims to modify the Wisconsin Card Sorting Test which assesses cognitive flexibility, into a military relevant map task. Thirty-four military officers from all service branches completed the map task. 
 
 
 
 
Design/methodology/approach 
 
 
 
 
The purpose of this study was to modify a current psychological task that measures cognitive flexibility into a military relevant task that includes the challenge of overcoming experiential bias, and understand underlying causes of individual variability in the decision-making and cognitive flexibility behavior of active duty military officers on this task. 
 
 
 
 
Findings 
 
 
 
 
Results indicated that non-perseverative errors were a strong predictor of cognitive flexibility performance on the map task. Decomposition of non-perseverative error into efficient errors and random errors revealed that participants who did not complete the map task changed their sorting strategy too soon within a series, resulting in a high quantity of random errors. 
 
 
 
 
Originality/value 
 
 
 
 
This study serves as the first step in customizing cognitive psychological tests for a military purpose and understanding why some military participants show poor cognitive flexibility.

into a military-relevant map task. Thirty-four military officers, from all service branches, completed the map task. The purpose of this study was to (1) modify a current psychological task that measures cognitive flexibility into a military-relevant task, and (2) understand the underlying causes of individual variability in the decision making and cognitive flexibility behavior of active duty military officers on this task. Results indicated that nonperseverative errors were a strong predictor of cognitive flexibility performance on the map task. Decomposition of nonperseverative error into efficient errors and random errors revealed that participants who did not complete the map task changed their sorting strategy too soon within a series, resulting in a high quantity of random errors. This study serves as the first step in customizing cognitive psychological  Mentally agile leaders are able to anticipate and adapt to a given situation in order to make the best decision U.S. Department of the Army, Training and Doctrine Command, 2012). For example, the type of operations executed in Iraq and Afghanistan required military leaders to make daily assessments of the situation in their environment and make the necessary changes to their tactics for survival (Brown, 2007;Hartman, 2008;Mulbury, 2007). In psychology and neuroscience, this ability is known as cognitive flexibility and has been tested in multiple laboratory environments (Vartanian & Mandel, 2011). Although there are laboratory-based tests that measure cognitive flexibility, they are not directly applicable for military training needs (Perla, 1990). Thus, the purpose of this study was to (1) modify a current psychological task that measures cognitive flexibility into a military relevant task, and (2) understand underlying causes of individual variability in the cognitive flexibility behavior of active duty military officers on this task.
One common psychological task of cognitive flexibility is the Wisconsin Card Sorting Task (WCST) (Grant & Berg, 1948). The WCST taps the working memory, set switching, and inhibition components of executive function. Participants view five cards, with one card displayed at the top center of the computer screen and the remaining four displayed across the bottom of the computer screen. Each card contains symbols that vary in number, shape, and color. Over several trials, participants try to figure out the matching rule that will correctly match the card on top of the screen with one of the four cards at the bottom of the screen. Unknown to participants, the matching rule changes once they have 10 consecutive correct matches. For example, after 10 consecutive correct matches based on the color of the symbols, the matching rule could then change to the number or shape of the symbols. Thus, participants must not only learn and maintain in working memory the correct matching rule while inhibiting irrelevant stimuli, but also exhibit cognitive flexibility in detecting when the rule has changed and adapt their selections accordingly (Grant & Berg, 1948). The task is completed when participants either successfully complete two rounds of each matching rule for a total of six rules, or until they complete 128 trials. Successful performance on the WCST requires both set switching (switching to a new sorting rule, based on feedback) and set maintenance (maintaining the appropriate strategy long enough to reach the next sorting rule) (Barceló & Knight, 2002;Huizinga & van der Molen, 2007;Miyake et al., 2000).
Based on these findings, we examined cognitive flexibility performance in terms of both set maintenance and set switching.

B. MAP TASK (MODIFIED WCST)
The map task was developed in consultation with military advisors. On a computer screen, participants saw five maps, in which one map is at the top center of the screen and the remaining four are across the bottom of the screen (see Figure 1). Each map contains military graphics that vary in meaning, color and shape (U.S. Department of the Army, 2004). Graphics have three different categories distinguishable by their color: friendly force (blue), type of intended action (e.g., ambush; black), and type of enemy force (red). Each of these categories has three different possible shapes and each shape indicates a particular type of friendly force (rectangle and circle), intended action (lines and arrows), or enemy force (diamond) (see Figure 2). Similar to the method of Nelson (1976), we reduced the matching criteria on the map task to the type of graphic: friendly, intent, or enemy. For example, if the current matching rule is friendly graphics and the top map shown is similar to the card in Figure 1, then the correct choice would be to choose the map in the lower left-hand corner of Figure 1. One additional modification is that not all maps have all three types of graphics and participants can match maps based on the absence of graphic type (see Figure 2).  or more consecutive correct trials without completing that rule. Similar to the method of Barceló and Knight (2002), we decomposed nonperseverative errors into efficient and random components as indices of set switching and set maintenance (Huizinga & van der Molen, 2007). Set switching is indexed by perseverative errors and efficient errors, in which fewer perseverative errors and greater efficient errors indicates better set switching. Set maintenance is indexed by random errors, in which fewer random errors indicate better set maintenance (Huizinga & van der Molen (2007). Efficient errors are scored when an incorrect response is given during the second trial of a new matching rule series. Random errors are an incorrect response on a trial after the participant achieved a correct response on the previous trial (Barceló & Knight, 2002).

Demographic Survey
Age, gender, service branch, rank, and deployment experience were captured in the demographic survey.

Posttask Survey
Participants completed a free response question regarding the map feature on which they sorted and an ordinal scale question regarding how quickly they realized the sorting rule had changed: immediately/after 1-2 trials; after a few trials (3-4 trials); after several trials (5+ trials); and did not realize the sorting rule had changed.

Trails A and B
Because the map task places demands on visual processing speed, we included Trails A and B tests as covariate measures (Wechsler, 2008). In Trails A, the numbers 1-25 are randomly distributed on a worksheet. The participant starts at 1 and must draw a line to each number in numerical order. Participants are instructed to work as quickly and accurately as they can. In Trails B, participants now see both numbers and letters and must connect 1 to A, A to 2, 2 to B, and so on until they reach L, then 12. They also are instructed to work as quickly and accurately as they can. The test retest reliability on these measures ranges from 0.76 to 0.94 (Wagner, Helmreich, Dahmen, Klaus, & Tadic, 2011). In the current sample, performance on Trails A and B was moderately correlated.
Trails A and B have age-and education-based norms; these norms were used in computing Trails A and B performance in the current sample (Tombaugh, 2004).

Digit Span Forwards and Backwards
Because the map task also relies on working memory, a digit span forwards and backwards test from the Wechsler Adult Intelligence Scale (WAIS-IV) also were included as covariates (Wechsler, 2008). In digit span forwards, the experimenter states a series of digits, starting with two digits, and the participant must repeat them back. Then, the number of digits increases, with two trials per number of digits. The test is discontinued if the participant has an incorrect response to both trials for a particular number of digits or reaches the maximum of eight digits. In digit span backwards, the same procedure is followed, except this time the participant must repeat the digits in the reverse order, up to a maximum of eight digits. Test retest reliability of the digit span measures range from 0.66 to 0.89 (Lezak, 1995). In the current sample, performance on digit span forwards and backwards was positively correlated.

D. STATISTICAL MODELING TECHNIQUES
A combination of factor analysis and k-means clustering was utilized to determine if distinct groups of performers existed. Factor analysis indicated that nonperseverative error was the highest loading variable, with a value of 0.99. Next, cluster analysis produced three distinct groupings based on nonperseverative error score: the low error cluster (n = 8), moderate error cluster (n = 12), and high error cluster (n = 14) (see

III. PROCEDURES
The Naval Postgraduate School's Institutional Review Board approved the study.
Participants attended the laboratory individually for a single testing session. They first completed the approved consent form, then the demographic survey, Trails A and B, and Digit Span tests. Participants then sat at a standard desk and completed the computerized map task as if they were informing, yet removed from, tactical operations from a military operations center. Finally, participants completed the posttask survey questionnaire. Table 1 shows the summary statistics of the map task results. Performance on most measures is consistent with results from the original WCST, including a sample of veterans of similar age to our participants (Shan, Chen, & Su, 2008;Shura, Miskey, Rowland, Yoash-Gantz, & Denning, 2015). However, the percentage of nonperseverative error was higher than that of previous studies (Shan et al., 2008: a mean of 20.86% [sd = 13.57%]).

A. CLUSTER ANALYSIS RESULTS
Cluster analysis revealed three clusters of participants based on the percentage of nonperseverative errors. All participants in the low-error cluster, one participant in the moderate-error cluster, and no participants in the high-error cluster completed all six-rule changes of the map task. Figure 4b shows the number of trials that participants required to complete the first matching rule clustered by total nonperseverative errors. All participants in the low-and moderate-error clusters, and only 10 of the 14 participants in the high-error cluster, completed the first matching rule. Furthermore, Figure 4c indicates that all participants in the low-and moderate-error clusters, and only 2 of the 14 participants in the high-error cluster completed the first three matching rules. Figure 4d displays that all 8 participants in the low-error cluster, 6 of the 12 participants in the moderate-error cluster, and only 1 of the 14 participants in the high-error cluster completed the first five matching rules.
Figures 4a-4d. Strip charts displaying the distribution of map task participants clustered by (a) nonperseverative error, (b) number of trials to complete 1st rule, (c) number of trials to complete 3rd rule, and (d) number of trials to complete 5th rule. The x-axis represents the total number of nonperseverative errors, and the y-axis represents the cluster label for a group of participants.
We further classified participants as high or low performers. High performers completed all six rule changes; low performers completed five or fewer rule changes. All but one high performer was categorized into the low-error cluster group. The nine high performers had a total number of nonperseverative errors that were significantly lower than that of low performers. As expected, the high performers needed fewer trials to complete the first matching rule than did the lower performers (M = 53.72, z = -3.4, p < 0.002, effect size = 0.583). Analysis of the failure to maintain set metric produced an insignificant difference on this measure between high and low performers.

B. NONPERSEVERATIVE ERROR ANALYSIS
We next sought to better understand the variability in the nonperseverative error rate. No significant associations were found between the nonperseverative error rate with 4a 4d 4b 4c performance on digit span and Trails A and B tests, nor with ground combat experience (all p's > 0.22). Next, we decomposed nonperseverative errors into efficient or random errors (see Table 2). As expected, high performers achieved a change in the matching rule efficiently, whereas low performers shifted to a new rule too soon in the current series. Although there was no significant difference between high and low performers in the average number of efficient errors, high performers had, on average, significantly fewer perseverative errors (z = -3.27, p = 0.0006, effect size = 0.56) and random errors.
Thus, high performers had better set maintenance and set switching than low performers. Note: Efficient errors occur when an incorrect response is given during the second trial of a new matching rule series; more efficient errors indicate better set switching. Random errors are an incorrect response on a trial after the participant achieved a correct response on the previous trial; fewer random errors indicate better set maintenance (Barceló & Knight, 2002).

C. POSTTASK SURVEY
We also analyzed the responses of participants on the posttask survey, based on their cluster groupings. Variability in self-reported realization of rule change increased from the low-error to high-error cluster group. All low-error cluster participants reported that they realized a rule change within 1-2 trials. The moderate-error cluster reported the following: four participants realized a rule change within 1-2 trials, seven participants realized a rule change within 3-4 trials, and one participant reported noticing a rule change in 5+ trials. For the high-error cluster group: three participants reported realizing a rule change within 1-2 trials, three participants within 3-4 trials, four participants within 5+ trials, and four participants did not realize the rules changed at all.

V. DISCUSSION
Military operations require leaders to have agile and adaptive decision-making skills. However, current military training typically does not focus on training the cognitive functions necessary for optimal decision making, such as cognitive flexibility.
The purpose of this study was to create a military-relevant measure of cognitive flexibility and to understand the underlying mechanisms in variability in cognitive flexibility performance that could aid military decision-making training.
Towards meeting these goals, we modified the WCST into the military-relevant map task. Although adequate performance was met in terms of percentage of correct responses and number of rules obtained, we were surprised by the high frequency of nonperservative errors compared to studies using the original WCST (Nelson, 1976;Ozonoff, 1995;Barceló & Knight, 2002;Kado et al., 2012). The high frequency of nonperseverative errors cannot be explained by poor working memory or processing speed. One possible explanation is that, contrary to the original WCST, the military symbols on the map task have a specific meaning and experienced officers could read each card as a military operation. The symbols on the map task are primarily groundbased and this could result in officers familiar with these symbols attempting to match the cards as a type of military operation, and not just simply matching on the correct symbol color. However, no significant difference in nonperseverative error rate was found between participants who previously had a ground combat deployment and those that did not. Future data collection will entail collecting additional information about subjects' ground-based military operations experience to test this idea.
Using cluster analysis, we determined that the number of nonperseverative errors may be a useful assessment tool of cognitive flexibility for Soldiers, as all participants neatly grouped into one of three clusters (high, moderate, and low nonperseverative error rates) and because the nonperseverative error rate was highly correlated with performance. Indeed, the low-error cluster captured all but one high performer.
Additionally, decomposing nonperseverative errors into efficient and random errors produced further insight into the potential reasons why some participants did not complete all six matching rules of the map task. Participants that achieved all matching rules showed a consistent pattern in which they explored the different options early in the sorting series to determine the new matching rule and then continued to make selections that met that particular matching rule until it changed. Thus, these participants demonstrated adequate cognitive flexibility by exhibiting both set switching and set maintenance (Huizinga & van der Molen, 2007). In contrast, low performers showed especially poor set maintenance, as indicated by a high quantity of random errors. The end result of this pattern is that these officers are switching decision-making tactics too soon to see if a particular tactic actually works.
The implications of the results are that (1) set maintenance may be a skill that is currently undertrained among military officers and (2) cluster analysis by nonperseverative error rate is a parsimonious method for identifying officers who require additional cognitive flexibility training. This study was a preliminary attempt at measuring military cognitive flexibility and participants were primarily officers.
Therefore, additional studies examining cognitive flexibility among a wider range of military personnel are required to provide support of these initial findings.