Who writes and who responds? Gender and race-based differences in open annotations

Abstract

Purpose

Historically minoritized students are routinely silenced in classroom settings. This study aims to explore whether open annotations encourage students with historically minoritized gender/racial/ethnic identities to share knowledge and ideas. In addition, this study explores how the intersectionality of student identities relates to their experiences of open annotation and assess gendered and racialized achievement of student learning objectives.

Design/methodology/approach

This study uses mixed methods to examine the use of an open annotation tool (Perusall) to foster the redistribution of epistemic authority and more equitable interactions in science, technology, engineering, and math and humanities courses at the intersection of environmental and social justice issues. The study design draws on illustrative case study methods to assist others in seeing the potential and considerations in using a similar pedagogical approach.

Findings

An open annotation tool like Perusall can foster more equitable interactions for historically minoritized students. Women reported that open annotations deepened knowledge and engagement with the source and their peers. Women of color, in particular, acknowledged the benefits of social annotations as a tool that redistributes epistemic authority. Conversely, men were more likely to comment on dissatisfaction with grading.

Originality/value

This study suggests the value of open annotation as an effective and accessible method to foster inclusive classrooms. Through examining epistemic authority in social annotations, this study provides a novel approach to addressing the disengagement of historically minoritized students.

Citation

Bakermans, M., Pfeifer, G., San Martín, W. and LeChasseur, K. (2022), "Who writes and who responds? Gender and race-based differences in open annotations", Journal for Multicultural Education, Vol. 16 No. 5, pp. 508-521. https://doi.org/10.1108/JME-12-2021-0232

Introduction

A large body of literature examines the importance of student participation in classroom discussions and various active learning activities (Shah et al., 2015; Sereno et al., 2020). Much of this literature points to the benefits of such participation in learning gains and skills development for students and faculty members (Gillis, 2019). However, there are multiple reasons that students may not actively participate in classroom discussions, including lack of confidence, low commitment to the course, insufficient preparation for a given class and differences in reading strategies (Rocca, 2010; Carillo, 2017; Leraas et al., 2018).

Although these are notable influences on who engages in a classroom discussion about texts, this study does not focus on these mechanisms. Instead, this study focuses on how students enter classrooms with preexisting beliefs and assumptions about who has the greatest knowledge and the right to speak, called epistemic authority (Sayles-Hannon, 2012). Epistemic authority is often accorded to certain students in raced and gendered ways such that students who are seen as white and male are accorded a surplus of epistemic authority, and women and historically minoritized students are seen as lacking (Sayles-Hannon, 2012) These beliefs and assumptions lead to a double-edged sword of behaviors: students who believe they have low epistemic authority themselves stay silent; at the same time, those with a high sense of their own epistemic authority not only engage but often presume their engagement takes precedence over those they see as having relatively little epistemic authority.

Students’ epistemic authority, or lack thereof, is not inevitable nor entirely determined through personality and identity politics. The uneven distribution of epistemic authority in the classroom often reflects more structural socio-cultural and economic inequalities. Classroom climate, thus, plays an important role in who feels able to participate in the discussion and who does not. Despite gains in recent decades in admissions to Science, Technology, Engineering, and Mathematics (STEM) degree pathways, retention and persistence to degree, a “chilly climate” persists for some students, predominantly women and students of color (Hall and Sandler, 1982; Laufer, 2012; Lee and McCabe, 2021). Instructors can unintentionally reinforce these problems in various ways, from continuing to only call on vocal students to not carefully considering inclusivity in the classroom (Lee and McCabe, 2021). When these problems persist, learning gains for all students are diminished.

One way faculty can create classrooms inclusive of historically minoritized student voices is to think about avenues beyond the spontaneous raising of hands or open dialogue. For example, faculty may use strategies to expand the space for engaging that allows all students to participate in the discussion; such strategies are designed to open participation by removing space for the loudest voices to dominate. In particular, open educational pedagogies and practices can center each student in creating, managing and sharing knowledge (Cronin and MacLaren, 2018; Koseoglu and Bozkurt, 2018; Lambert and Czerniewicz, 2020). These practices can vary widely, from an open syllabus to creating open education resources and with different degrees of openness (DeRosa and Jhangiani, 2018; Bali et al., 2020).

Social annotation is one open tool that allows users to write annotations on any electronic resource, visible and interactive with others. Social annotation thereby turns the solitary act of engaging with a source into a collaborative learning experience (Novak et al., 2012). Social annotation is often used in the classroom to facilitate comprehension and engagement with sources (Carillo, 2017; Brown and Croft, 2020). Using this type of open pedagogy has been demonstrated to improve student learning (Lin and Lai, 2013; Suhre et al., 2019), increase reading and peer-critique skills (Johnson et al., 2010; Carillo, 2017), facilitate group inquiry learning (Li et al., 2015), develop community and communication among peers (Zhao et al., 2018; Adams and Wilson, 2020) and provide new perspectives to participants (Novak et al., 2012).

Social annotation tools and platforms are gaining more exposure and growth in digital learning and include but are not limited to Annotation Studio, Diigo, Hypothesis, NowComment and Perusall (Novak et al., 2012; Ghadirian et al., 2018). Social annotation tools range in openness, design and flexibility (Novak et al., 2012; Seatter, 2019; Marissa, 2021), and instructors should carefully consider features when selecting an annotation tool for the classroom. Some factors to consider are differences in setting (public or private), ease of navigation, annotation options, accessibility, types of sources (e.g. pdf, webpages, videos), additional student capabilities (e.g. upload their materials and note-taking), need and integration with your institution’s Learning Management System (LMS) and support (Novak et al., 2012). For example, we used Perusall as an open annotation tool because it allowed multiple types of sources (e.g. videos, webpages and documents), provided summary and engagement information for each student (including adjustable scoring), and was integrated into our institution’s LMS.

Regardless of which tool is used, social annotation can be a student-centric, process-focused practice that can include a social justice component if the source discusses these issues or gives a voice to marginalized authors (Bali et al., 2020). However, few instructors consider whether open annotation can center equity in the classroom (Brown and Croft, 2020). Indeed, the process of annotating and asserting one’s knowledge and perspective can express power and can contribute to deconstructing norms around who has the authority to produce and convey knowledge (Kalir, 2020). Unfortunately, open practices do not guarantee equity, and online tools can expose participants to othering and harassment (Phirangee and Malec, 2017; Ortega et al., 2018; Brown and Croft, 2020). This study examines the use of an open annotation tool (Perusall) to foster inclusive pedagogy and redistribute epistemic authority in undergraduate STEM and humanities courses.

Methods

This study used mixed methods to examine whether an open annotation tool can provide access to more equitable gender and race/ethnicity-based dynamics in STEM and humanities courses at the intersection of environmental and social justice issues. The study design draws on illustrative case study methods (Yin, 2009) to assist others in seeing the potential and considerations in using a similar pedagogical approach. Case studies are particularly appropriate when the boundary between a phenomenon of interest, such as a particular pedagogical approach, and its context is not well-defined (Yin, 2009). Given the influence of context on how marginalized students show up in the classroom, a case study is appropriate. The choice of an illustrative design, rather than explanatory, aligns with the goals of this study: to examine and share the benefits, challenges, and strategies involved in using open education tools and pedagogies to support marginalized students and close equity gaps in undergraduate courses.

Examination of this case is organized around the following research questions:

RQ1.

Were there gender and race/ethnicity-based differences in the achievement of student learning objectives related to collaboration, integration of new ideas and critical analysis of sources among students using open annotation? If so, what were they?

RQ2.

To what extent did students experience the benefits and drawbacks of the open annotation tool?

RQ3.

What were the gender differences in student-reported experiences?

RQ4.

What were the race/ethnicity-based differences in student-reported experiences?

Case selection

The case is illustrated within the context of Worcester Polytechnic Institute (WPI) at a time when providing better access and sustained pathways to STEM for women has been an intentional focus. As a result, WPI is now among the most gender-balanced STEM institutions in the USA. The incoming Class of 2021 is 44% women, following a decade of dedicated recruitment efforts and steady increases in the number of women who matriculate. Efforts to retain women in STEM pathways include dedicated scholarships, a new center devoted to supporting equitable teamwork, and a new Gender, Sexuality and Women’s Studies Program.

Although the university has prioritized recruitment and retention of women, strategies to support gender-inclusive pedagogies within courses have been more emergent and diffuse. Four courses bound the case; all focused on environmental and social justice issues, emphasizing integrating science and humanities disciplinary perspectives: Great Problems Seminar – Extinctions, Introduction to Environmental History, Biodiversity, and Topics in Environmental History – Race and Justice. This study illustrates the efforts of three faculty members teaching these courses to self-organize around an annotation tool, Perusall (https://app.perusall.com/), to ask students to engage in learning with fundamentally different classroom dynamics.

This tool allows students to access, read and annotate all sources (e.g. readings, videos) in an open and shared manner to change the solitary act of reading into a collaborative exercise. While students are engaging with a source, we guided them to identify the key ideas, assumptions and gaps in knowledge, and conclusions; trace the development of arguments; expand on information and provide additional data or examples; connect ideas and information to knowledge from other sources (e.g. readings, discussions); and clarify passages (e.g. terms, concepts) for yourself and others. Furthermore, we instructed students to engage with each other by asking questions that can build a conversation and deepen learning; share answers, perspectives and external knowledge; and upvote questions or answers they find most helpful to their understanding of the content.

One noteworthy difference in how Perusall was used as a tool across courses, in this case, was the use of the grading functionality. The three faculty members included in the case did not organize a unified strategy for using the built-in ability for Perusall to provide auto-grading; rather, the faculty each explored this functionality to learn about the potential benefits and constraints afforded through auto-grading. While auto-grading has presented challenges when using open annotations (Cecchinato and Foschi, 2020), allowing faculty to choose how to use this potential tool in their course is aligned with academic freedom and is an authentic problem faced by those using these tools. All three faculty adjusted the preset scoring options in Perusall and allowed scoring to accumulate so students had multiple ways to earn full credit.

This case, therefore, serves as an illustration of what is possible within an institution that is supportive of innovative classroom approaches to achieve inclusivity and equitable learning outcomes. Additionally, because open annotations are an online tool and can be used in various contexts that span from in-class to hybrid or fully online learning, this case can inform improving inclusive pedagogies across these contexts.

Participants

A total of 134 students (all students enrolled in the four courses in this study) participated in the study. Of these, 113 undergraduate students (84%) completed the survey, including 46% men, 53% women and 1% nonbinary (self-disclosed agender). Within the survey sample, 78% are white, and 22% are historically racial/ethnic minoritized students (13% Asian/Pacific Islander, 6% Hispanic/Latino, 2% black/African American, 1% Native American, 1% other, not disclosed). The majority of students in the sample were in their first year (62%), alongside 21% second-year students, 12% third-year students and 4% fourth-year students.

Data collection

Two sets of data were examined for the analyses in this study; all data were collected with institutional IRB approval (IRB-21–0021).

Student learning outcomes survey

In the final week of each course, students were asked to complete a survey with the same items across the four courses. The question stem asks, “Working through Perusall in annotating sources helped me […]” for a set of eight outcomes. These were based on the Association of American Colleges and Universities (AAC&U) Reading VALUE Rubric (AAC&U, 2009), including comprehension, genres, relationship to text, analysis, interpretation and reader’s voice. In addition, we added two outcomes on collaborative learning from peers and feeling a part of a collaborative learning community. Students could respond on a Likert scale from 1 (strongly disagree) to 5 (strongly agree). The survey collected student demographics.

Student reflections

In addition to items assessing student learning, the survey included an open-ended prompt asking students whether they had any additional thoughts they would like to share about Perusall and the annotation process. These qualitative data were included to understand what rose to the level of priority in students’ experiences.

Analysis

Analysis proceeded in three stages. To address the first research question, differences in student learning outcomes across courses were examined using analyses of one-way variance to determine whether data could be pooled for further analysis. Student learning outcomes in quantitative survey data were then analyzed using independent sample t-tests to contrast students who self-identified as men with those who self-identified as women, and students who self-identified as white and those who self-identified as any other race/ethnicity. This organization of multiple historically minoritized races/ethnicities was used to establish sufficient sample sizes for these analyses. Thus, the analyses to assess whether there were any gender- or ethnicity-based differences in the achievement of student learning outcomes were entirely quantitative.

To address the second research question, we used both qualitative and quantitative analytical procedures. Qualitative data were thematically coded using open and axial coding (Clarke and Braun, 2014). Student reflections were coded without their demographic identifiers to allow emergent coding to be completed without interference or bias.

After all qualitative data were coded, the proportion of students with a code present was calculated for each demographic group. This third and final stage of analysis allowed us to assess whether any demographic groups disproportionately reported particular experiences compared to their representation in the full sample. By assessing quantitative trends in qualitatively coded experiences, we established how widespread experiences were across demographic groups.

We then illustrate these experiences using quotes from student reflections. With each claim, we ground the justification in a quantified trend and then provide several illustrative examples of how individual students experienced that trend. Analyses do not generalize from individual experiences to populations sharing similar demographics but seek to provide examples of how broader themes shared across disproportionate groups of students were described directly in the data.

Findings

Role of open annotating in learning

Students were asked to assess the extent to which Perusall helped them meet student learning objectives within three broad categories: collaboration, integrating new ideas and critical analysis of sources. On average, students reported that Perusall strongly helped them collaborate, somewhat-to-strongly helped students integrate new ideas and somewhat helped them critically analyze sources (see Table 1). There were no patterns of significant differences across courses (see Table 1); we, therefore, pooled data across courses to examine gender differences.

In this study, women and men using Perusall reported similar usefulness of the tool across all eight learning outcomes related to collaboration, integrating new ideas and critical analysis of sources (see Table 2). If open annotation were not successful in creating a virtual classroom environment where all students felt they could voice their perspectives and contribute to the collaborative knowledge being built, we would expect to see significant differences; according to the extant body of knowledge, women would have rated the utility of this mechanism lower than their men classmates, whose voices are privileged in a traditional in-class discussion.

Similarly, white students and historically minoritized students reported comparable usefulness of the tool for all learning outcomes (see Table 3).

Student experiences of the tool’s potential

The survey asked students whether they had introduced people outside of the course (e.g. roommates, peers, friends) to Perusall as a potential tool to use or explain its use for their work. While only 17% of students indicated that they had, the vast majority (74%) of these were women. Historically minoritized students were also more likely to have shared the tool. Although historically minoritized students account for only 22% of the survey respondents, they make up 37% of those who indicated they had shared the tool with peers. Although a blunt indicator, this suggests that women and historically minoritized students may perceive the potential value of the tool for their learning differently than those typically privileged in class discussion.

To further investigate, we coded open response comments in the survey to elicit themes related to student concerns and student perspectives of the tool’s value. Students were asked in the survey whether they had any additional thoughts they would like to share about using Perusall; 60% of students wrote responses. Two themes emerged. First, 38% of comments (from 20% of the students in the study) indicated ways that Perusall deepened their learning compared to other approaches to course readings and class discussion. Second, 19% of comments (from 10% of students in the sample) indicated concerns over how their contribution was graded. We further coded students’ reflective statements to make sense of whether and how themes were clustered within and across student identities.

Gendered understandings of deeper learning through Perusall

The majority (69%) of comments that discussed how Perusall did or did not support deeper learning were made by women. Some students indicated that Perusall allowed them to engage the text with the added perspective of their classmates’ interpretations. As one representative woman noted:

Perusall was a good tool to better learn how to annotate and understand a certain topic more […] I would make annotations, ask questions, and answer questions and then go back at a later time to see what everyone else has written […].

Other students also noted how helpful it was to see everyone else’s thoughts while reading the assigned texts. For example, “I enjoyed being able to see other people’s comments and learn from them. Often, I did not think of the points they made, so it was very helpful to be informed of new perspectives and opinions.” Women were more likely than men to make statements that they valued their classmates’ multiple perspectives. The practice of annotating texts can foster the consideration and comparison of multiple perspectives (Carillo, 2017).

In sharing how Perusall influenced their learning, several students provided feedback that might assist faculty in framing student expectations about college-level work. For example, many students were shocked at how much time it took to read the assigned readings and process what they were reading (what might be considered “reading for understanding”), rather than simply skimming a text to assimilate new facts. As one man shared:

I feel as though Perusall and the ability to annotate sources collaboratively was a useful tool but I was not able to take complete advantage of it. It helped me with annotations as it was not something I was adept in previously but I also found that my annotations did not have much of a purpose after initially making them. If I did make an annotation it was lost after my initial reading and I never found the time to go back and assess what the annotations greater meaning could be. I often found the reading for the class became a secondary assignment. The readings were interesting but could feel time consuming and annotating could take too much time while I still had other work to complete. As a result of this my annotations often lacked even though I had thoroughly read the article.

The above passage is one example of several students who wrote about the annotation process seeming to detract from their ability to get all of their work done, suggesting that this level of process is not part of basic requirements. Positioning it as “extra” effort is an indication of student expectations regarding college coursework.

With the right framing, Perusall might be used as a tool to assist students in adjusting their expectations of what reading to understand involves and how time-consuming it should be. This may be particularly relevant for students in STEM programs whose experiences largely reify expectations that reading is intended to transfer technical knowledge rather than to spark deep thought of multiple, nuanced considerations. In the present study, three of the four courses examined provide course credit in humanities in interdisciplinary courses for STEM students.

Intersectional experiences of deeper learning through Perusall

Women of color in our case study were disproportionately more likely to reflect on deeper learning in their comments on using open annotations than either white women or men of any race/ethnicity. Although 60% of the historically minoritized students who provided reflections on Perusall are women, 100% of those who discussed the tool’s potential to support deeper learning were women. These women of color used Perusall in three of the four courses studied, not clustered within a single course experience.

Women of color often noted that open annotating provided ways of accessing a broader range of viewpoints and perspectives. As one woman of color remarked:

I enjoyed being able to see other people’s comments and learn from them. Often, I did not think of the points they made, so it was very helpful to be informed of new perspectives and opinons.

Another woman of color connected learning about others’ insights with her own increased engagement in the source materials:

It was really helpful and was a way to keep me engaged in the reading. Many peers had very interesting things to say, and I learned far more than what was on the actual textbook.

Every woman of color who commented on whether and how open annotating engaged them in deeper learning made similar points about how social sharing exposed them to new perspectives in useful ways. As textbooks are often dominated by white voices, the increased access to peers’ knowledge provides a more diverse academic conversation. Without exception, women of color expressed valuing the ability to connect with a diversity of viewpoints and positioned that exposure as beneficial to their own learning. For students of color, accessing how other students of color think about the same concepts is part of the social justice potential of open, social annotation. In addition, the emphasis on learning from others’ perspectives and opinions suggests the possibility of social annotation to provide avenues that foster intercultural competencies and skills, which are critical to addressing social justice in the classroom and promoting cross-cultural education.

A few students also noted how the tool might change who feels like they can contribute to class discussion. As one student shared:

In other humanities classes I have taken, I personally prefer just reading the documents before class on Canvas. One advantage while using Perusall is for grading participation because some people prefer not to talk as much in class.

Particularly notable is that this woman of color recognized the advantage of open annotating for those who do not regularly talk in class, despite stating that this is not her preference. This and similar statements point to how social annotation can provide paths for the redistribution of epistemic authority and the expansion of participation grading criteria for learners that do not usually fit traditional standards of participation (i.e. talking in class). It is important to note that women of color tend to acknowledge and report the benefits of social annotation as a tool to redistribute and democratize epistemic authority more than other groups analyzed in this study.

In contrast, white women tended to note how they were able to leverage classmates’ contributions to their own learning process. Although social annotations benefiting an individual’s learning process is a desirable outcome, it is essential to note that when white women benefit from students of color, the social annotation reifies power dynamics that commodify students of colors’ intellectual labor for white students’ benefit. This strategy sometimes involved direct tactics for using the tool to meet their own learning needs. For example, one student shared:

I found Perusall very easy to use and vital to understand a text further than what the reading explicitly said. Having the ability to start conversations and ask questions right on the reading was extremely helpful when learning in this class.

This was a common tactic, which students found a primary benefit of social annotation and the Perusall tool. Another white woman student shared more detail on why this tool was helpful for her own learning and for learning from others.

It was awesome to be able to use this tool to add our own annotations and thoughts about the articles we read, but also being able to work off of other people ideas and read how other people interpret the articles and what their personal opinions are. As well as being able to hold an academic conversation within Perusall to be able to learn more not only about the specific articles for each assignment but about the articles overall and other content we covered in class because it was easy to make connections between other articles and concepts we read or learned about previously allowing for us to get a better understanding of the classes content overall and then applying that to our final project.

These and other white women in the study valued the diversity of opinions, analyses and interpretations shared through open annotations because they served their learning process and outcomes. While supporting individual learning processes is certainly a goal of using open annotation, there is a difference between valuing the contribution of diverse, often-silenced perspectives because it offers “new perspectives and opinions” from others and opens new forms of participation to people who “prefer not talk as much in class” (a collective learning value) and valuing the tool for its benefits for individual learning process or specific outcomes (self-centered learning value). Although fostering individual learning processes is a goal, the latter can be a problematic form of knowledge appropriation, which, in effect, might reproduce the raced and gendered dynamics of the chilly classroom that we intend to challenge. This difference between self-centered versus collective learning values as a form of knowledge appropriation can help expand research on epistemic positioning, primarily used in the study of professional academia, in the context of social annotation and undergraduate education (Bacevic, 2021).

These examples from our data show some of the limitations of only using open annotation to address injustice and epistemic disparities in the classroom. Such tools can offer opportunities to redistribute epistemic authority, but moving students with more historically privileged identities into a space where just epistemic collaborations and self-reflection are the norm requires careful work in helping all students confront how institutions and structures condition them to think of knowledge exchange and the benefits thereof (Bacevic, 2021; Long et al., 2020).

Grading concerns with Perusall

The emergent theme in student thoughts regarding grading did not come as a surprise. Students at this university tend to be ambitious, high achieving and overly preoccupied with grades. What is noteworthy in this theme is that those who used this opportunity to express concerns that Perusall involved unfair grading policies were disproportionately men. While Perursall’s automatic scoring was used differently in informing student’s final grades across courses (i.e. from only a suggestion on how to improve learning outcomes without direct influence on the student’s final grade to various degrees of inclusion as a percentage of the final grade), Perursall’s scoring emerged as a central theme, especially among men. Sixty-two percent of the comments concerned with grading came from men, compared to only 38% from women.

In several of the comments regarding grading, men stated that they did not receive credit commensurate with the quality of their contribution to the group’s collective knowledge-building. As one man shared:

I feel that Perusall did not take account to very thought based annotations but rather grades in the fact of the number of annotations I do. If I have a paragraph long explanation in Perusall deeply examining an idea, I don’t get as much points if I were to make short remarks or annotations.

Another man questioned Perusall’s valuation, stating:

I think sometimes the grade Perusall generates based on your annotations is at times too low and the software may grade annotations with great increasing difficulty as the size of the document on the software increases, as in the assignment to review our research dossier, I read through all of the articles and wrote over 70 comments, but Perusall still only gave me a generated grade of about 2.6 out of 3 (I believe if I remember correctly).

This student’s analysis suggests dissatisfaction with the quantification of grading annotations while also accepting that very premise to challenge the algorithm’s computations. This and other students often miss when parsing the grading is that the scoring includes more than the annotation itself: the scoring includes other engagement metrics related to active reading, responding to other students and having other students respond to them. The tension between self-assessing the quality of one’s contribution and wanting it to be easier to game the system was found widely within men’s feedback on grading.

To be clear, we do not see such efforts as counterproductive to social annotation’s learning outcomes. They are, as research shows, the long-term structural effect of letter- and number-based grading systems, which create preferences to seek out classroom tasks that will result in the desired grade on an assignment, rather than a focus on learning the material itself (Kohn, 2013). Our reporting on this result does not suggest that these students are intentionally gaming the system in ways that should be seen as immoral. As the research often shows, such behaviors are quite rational given the ways students are traditionally assessed. In highlighting this finding, we are pointing both to the existence of such things and that the auto-grading frustrates those structurally induced modes of engagement with course material.

Rather than locating the issue with grading externally (i.e. “the technicalities of the grading system are incorrect”), women who expressed grading concerns were far more likely to have internalized the issue. Women spoke of the pressure they felt rather than remarking simply on the injustice of a wrongly assigned lower grade. Part of the intention behind adopting open annotation tools is to encourage greater participation among those who are often disproportionately silenced in class discussion. The trend of women noting the pressure to engage more suggests that it works – the tactic might require efforts to mitigate unintended negative consequences rather than removing the pressure that prompts women to engage in the discussion.

A few women also noted the environment created through the grading policy. One woman shared that:

[…] [it] felt like being assigned to go ‘above and beyond’ in the discussions to get an A in the discussion made an environment where I just felt like I needed to make comments on like everything to get a grade rather than have organic discussion on the parts that I wanted to gain information on.

While men shared similar concerns in the same class and other courses, women explicitly understood the problem to be not just in the grading policy and not just for themselves but to be an issue of classroom climate. This suggests a fundamentally different set of interventions to inform a more productive classroom environment, rather than expressing a personal injustice that assumes the student has adequately self-assessed the value of their contribution.

Implications

As can be seen from our findings, adopting an open annotation tool like Perusall can be a helpful strategy for creating a classroom climate that is less chilly for women and other historically minoritized students, and it can help begin to work to redistribute epistemic authority in a more equitable and realistic way.

Using an open annotation tool like Perusall may provide a platform for historically minoritized students to become more visible in classroom discussion, which is particularly important in a majority-white context, like our courses and campus. At the same time, this pedagogical choice also encourages broader participation for all students. Social annotation offers a less intimidating space than typical classroom discussion settings by removing the traditional barriers put in place by implicit biases and other practices that favor certain students at the expense of others.

Our study also shows that while some students are frustrated at how auto-grading makes it more difficult to game the system, others find multiple benefits in seeing the thoughts of a broader set of their peers than in more traditional settings. Accessing a broader array of their classmates’ internal processing of the readings allows students to engage with them around the ideas in the readings and might provide important avenues to foster intercultural competencies and skills. Like many students in our study self-report, this, in turn, leads to deepened knowledge and engagement with the material in ways that mirror the larger literature on the benefits of participation in class discussion referenced at the outset of this article (Gillis, 2019).

In many ways, we treated open annotation as one strategy for addressing what we know are gendered and raced experiences of engaging in classroom discussion around assigned readings. In this way, open annotation tools, like Perusall, offer an entry point in rethinking the classroom space that is easy to adopt for faculty who may not be well-versed in inclusive pedagogies and less comfortable than others with implementing such changes. As noted above, we want to stress that adopting an open annotation tool like Perusall should not be seen as a solution to the problems associated with chilly classroom climates and issues of who is accorded epistemic authority and who is not. Still, it can be one way to illuminate structural issues in the distribution of epistemic authority in the classroom, begin to address them using focus open educational practices in a more socially just way, and offer new tools to open students to such changes (Lambert, 2018).

Although social annotation may offer opportunities, it can also pose a risk to marginalized student populations in the form of “colorblindness” and stereotypical assumption microaggressions (Phirangee and Malec, 2017; Brown and Croft, 2020). When implementing these tools in these courses, we took measures to reduce othering by including sources that explicitly validated the role of diverse knowledge systems, monitored annotations and participated in the threaded discussions and publicly recognized and affirmed diverse contributions and viewpoints (Brown and Croft, 2020). Therefore, research should situate this tool alongside other pedagogical moves designed to center historically marginalized students in ethical and responsible ways.

Furthermore, this study examined student perceptions of using this open annotation tool, and future studies need to explore differences in practice by students. Future analyses from the broader data collection for this case of open annotation will assess the various behaviors of engagement across gender and racial lines and unpack the roles of the college experience (via class year) and disciplinary expectations (via major) on intersectional experiences and participation. Future work will also triangulate student self-reports against student practices and products of annotating.

Table 1.

One-way analysis of variance in student learning outcomes by course, with mean (M) and standard deviations (SD)

	Course 1		Course 2		Course 3		Course 4
Student learning outcomes	M	SD	M	SD	M	SD	M	SD	F(3,109)
Collaboration
Engage in dialogue	4.24	0.74	4.59	0.69	4.39	0.74	4.45	0.69	1.25
Learn from peers	4.59	0.70	4.78	0.64	4.24	0.99	4.45	0.52	2.66
Feel part of a community	4.29	0.68	4.81	0.40	4.22	0.91	4.45	0.69	3.84*
Integrating new ideas
Relate ideas	4.12	0.64	4.59	0.75	4.34	0.86	4.64	0.51	2.62
Recognize broader issues	3.94	0.69	4.22	0.80	4.20	0.81	4.36	0.51	1.30
Critical consumption of sources
Discuss with an ethical disposition	3.82	0.76	4.19	0.74	4.12	0.87	4.18	0.75	1.39
Evaluate significance	3.79	0.73	4.07	0.87	3.73	1.03	4.36	0.67	2.04
Adjust reading strategies	3.71	0.84	4.04	0.71	3.80	1.03	4.27	0.79	1.56

Note:

*p < 0.05

Table 2.

Differences in student learning outcomes by gender, including mean (M) and standard deviation (SD) values

	Men		Women
Student learning outcomes	M	SD	M	SD	t	p
Collaboration
Engage in dialogue	4.31	0.85	4.47	0.60	−1.21	0.23
Learn from peers	4.37	0.91	4.61	0.72	−1.58	0.12
Feel part of a community	4.35	0.80	4.44	0.73	−0.61	0.55
Integrating new ideas
Relate ideas	4.27	0.84	4.46	0.68	−1.29	0.20
Recognize broader issues	4.08	0.81	4.24	0.65	−1.15	0.25
Critical consumption of sources
Discuss with an ethical disposition	4.02	0.85	4.07	0.76	−0.32	0.75
Evaluate significance	4.06	0.85	3.81	0.86	1.50	0.14
Adjust reading strategies	3.98	0.90	3.81	0.86	1.00	0.32

Notes:

n men = 52; n women = 59

Table 3.

Differences in student learning outcomes by racial/ethnic identities, with mean (M) and standard deviations (SD) included

	White students		Historically minoritized students
Student learning outcomes	M	SD	M	SD	t	p
Collaboration
Engage in dialogue	4.39	0.72	4.44	0.77	−0.30	0.77
Learn from peers	4.51	0.79	4.48	0.92	0.14	0.89
Feel part of a community	4.47	0.68	4.20	0.96	1.56	0.12
Integrating new ideas
Relate ideas	4.36	0.75	4.40	0.82	−0.25	0.80
Recognize broader issues	4.11	0.71	4.32	0.80	−1.24	0.22
Critical consumption of sources
Discuss with an ethical disposition	4.01	0.80	4.20	0.82	−1.03	0.30
Evaluate significance	3.94	0.84	3.80	1.00	0.72	0.48
Adjust reading strategies	3.92	0.82	3.80	1.04	0.60	0.55

Notes:

n white students = 87; n historically minoritized students students = 25