Abstract
Purpose
Wikipedia's inclusive editorial policy permits unrestricted participation, enabling individuals to contribute and disseminate their expertise while drawing upon a multitude of external sources. News media outlets constitute nearly one-third of all citations within Wikipedia. However, embracing such a radically open approach also poses the challenge of the potential introduction of biased content or viewpoints into Wikipedia. The authors conduct an investigation into the integrity of knowledge within Wikipedia, focusing on the dimensions of source political polarization and trustworthiness. Specifically, the authors delve into the conceivable presence of political polarization within the news media citations on Wikipedia, identify the factors that may influence such polarization within the Wikipedia ecosystem and scrutinize the correlation between political polarization in news media sources and the factual reliability of Wikipedia's content.
Design/methodology/approach
The authors conduct a descriptive and regression analysis, relying on Wikipedia Citations, a large-scale open dataset of nearly 30 million citations from English Wikipedia. Additionally, this dataset has been augmented with information obtained from the Media Bias Monitor (MBM) and the Media Bias Fact Check (MBFC).
Findings
The authors find a moderate yet significant liberal bias in the choice of news media sources across Wikipedia. Furthermore, the authors show that this effect persists when accounting for the factual reliability of the news media.
Originality/value
The results contribute to Wikipedia’s knowledge integrity agenda in suggesting that a systematic effort would help to better map potential biases in Wikipedia and find means to strengthen its neutral point of view policy.
Keywords
Citation
Yang, P. and Colavizza, G. (2024), "Polarization and reliability of news sources in Wikipedia", Online Information Review, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/OIR-02-2023-0084
Publisher
:Emerald Publishing Limited
Copyright © 2023, Puyu Yang and Giovanni Colavizza
License
Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
1. Introduction
Wikipedia is one of the most extensive encyclopedias worldwide, providing an open go-to reference for reliable online content and a key hub to the Web (Piccardi et al., 2020). Wikipedia’s articles are contributed by volunteers, following the policies of taking a neutral point of view (NPOV), verifiability of facts and sources and contributing no original research. In principle, all Wikipedia articles should be “based on reliable, independent, published sources with a reputation for fact-checking and accuracy” [1]. Sources are usually cited in footnotes and references. News media outlets provide a sizeable share of Wikipedia’s cited sources, yet they often contain both factual contents and opinions or viewpoints around them (Fetahu et al., 2015). News reporting from well-established outlets is generally considered reliable for statements of fact. However, a potential for viewpoint bias remains and may affect the integrity of knowledge in Wikipedia, or at least Wikipedia’s neutral point of view. While millions of volunteer contributors create and maintain free knowledge in Wikipedia (Aragón and Saez-Trumper, 2021), new challenges emerge in terms of information quality and reliability (Saez-Trumper, 2019; Morgan, 2019). What is more, news media often polarize around opposite political viewpoints (Patterson, 2011; Sutter, 2000). While previous work has focused on assessing the reliability of Wikipedia contents (Przybyla et al., 2022) and editors' possible biases (Rogers and Sendijarevic, 2012; Yasseri et al., 2013), researchers still have to systematically investigate Wikipedia’s knowledge integrity (Wikipedia, 2022; Sugandhika and Ahangama, 2022). We contribute to this line of work by exploring the political polarization of news media sources in Wikipedia. We further assess their reliability and determine whether there is a relationship between the two effects. To this end, we ask the following research questions:
Is there political polarization in the news media sources cited in Wikipedia?
What factors influence news media polarization in Wikipedia? Specifically, is there a relationship between news media political polarization and factual reliability?
In order to answer these questions, we rely on the large-scale dataset Wikipedia Citations (Singh et al., 2021); we use third-party sources to estimate the political polarization and reliability of news media outlets: the Media Bias Monitor (MBM) and the Media Bias Fact Check (MBCF). Following the approach taken by MBM, we consider political polarization across a mono-dimensional spectrum between liberal left and conservative right, acknowledging this as a limitation. Reliability conveys an estimate of the factual correctness of an outlet, in terms of contents and framing. We speculate that reliability and polarization might be related, for example with media outlets closer to a given political leaning being considered more reliable on average. Firstly, we provide a quantitative overview of news media sources' political polarization in Wikipedia (RQ1); secondly, we make use of regression analysis to clarify the relationship between news media source polarization, on the one hand, and an article’s topic, WikiProject and a source factual reliability, on the other hand (RQ2). Our aim is to inform Wikipedia’s knowledge integrity agenda (Taraborelli, 2019) by rising awareness on possible biasing effects in Wikipedia’s sources.
2. Previous work
2.1 Wikipedia’s core policies
Wikipedia strives to take a neutral viewpoint and provide reliable contents (Mesgari et al., 2015). To this end, Wikipedia abides to three core content policies:
Neutral Point of View (NPOV): “representing fairly, proportionately, and, as far as possible, without editorial bias, all of the significant views that have been published by reliable sources on a topic.” [2]
Verifiability: “other people using the encyclopedia can check that the information comes from a reliable source.” [3]
No original research: “Wikipedia articles must not contain original research.” [4]
These three policies could help to improve Wikipedia’s article quality (Pavalanathan et al., 2018) and could enable us to collectively address many of the practical issues stemming from collaboratively curating encyclopedic content (Arazy et al., 2006). However, from an epistemic perspective, they lay the responsibility for assessing content quality and reliability to third parties via reliable sources (Saez-Trumper, 2019). On the one hand, reliable sources are essential to Wikipedia’s status of a neutral encyclopedia, yet on the other hand the selection of sources invariably leads to controversies (Borra et al., 2014) and even edit wars (Sumi and Yasseri, 2011). To be sure, this might largely be a feature as some researchers believe that the existence of such controversies ultimately leads to better quality articles (Shi et al., 2019).
2.2 Knowledge integrity in Wikipedia
As one of the main repositories of free knowledge available today, Wikipedia plays a central role on the Web (Arazy et al., 2006; Smith, 2020). Its very widespread usage and radical openness to readers and contributors make Wikipedia vulnerable to malicious information attacks and disinformation (Saez-Trumper, 2019), which in turn could compromise Wikipedia’s knowledge integrity (Aragón and Saez-Trumper, 2021).
Knowledge integrity is one of the research priorities individuated by Wikimedia Research, whose aim is to identify and address threats to contents in Wikipedia, to increase the capabilities of patrollers, and to provide mechanisms for assessing the reliability of sources (Taraborelli, 2019). As of August 2022, Wikipedia is active in 318 language versions and each version is maintained by a dedicated (or language-specific) community [5]. While these communities have made substantial strides in enhancing the reliability of references within Wikipedia (Baigutanova et al., 2023a), challenges to knowledge integrity persist in various manifestations. For instance, some sources deemed unreliable in one language version may persist in articles across different languages (Baigutanova et al., 2023b). If we compare the size of the active editor communities with the scale of the Wikipedia project, it is clear that resources for patrolling and verifying contents remain on high demand (Morgan, 2019; Saez-Trumper, 2019). Besides, a lack of geographical diversity might favor nationalistic biases (Sato, 2021). From the perspective of contents, disputes between community members due to disagreements about the content of articles (Rogers and Sendijarevic, 2012; Yasseri et al., 2013), content verifiability (Lewoniewski et al., 2019; Redi et al., 2019) and quality (Lewoniewski et al., 2017; Rogers and Sendijarevic, 2012) are significant and enduring aspects of Wikipedia.
2.3 Wikipedia’s sources
The ‘verifiability’ policy guarantees the existence of an important aspect of Wikipedia: citations (Kaffee and Elsahar, 2021). Citations serve several important roles: “they uphold intellectual honesty and reduce the risk of plagiarism, they attribute prior work and ideas to their authors, they allow the reader to independently determine whether the referenced material supports the statements made by an editor in Wikipedia, and thus they help the reader gauge the strength and validity of the material an editor has relied on” [6]. However, evidence shows that references in Wikipedia are not too actively used by readers (Piccardi et al., 2020). In Wikipedia, scientific or scholarly literature takes up a large proportion of citations to sources (Nielsen et al., 2017; Singh et al., 2021), and Wikipedia’s citation rates are often aligned with those in the scholarly literature (Shuai et al., 2013; Mesgari et al., 2015; Yang and Colavizza, 2022a). Although it has been found that Wikipedia can have an influence on scientific research (Thompson and Hanley, 2018), and some professional journalists have also begun to use Wikipedia in their work (Messner and South, 2011), the debate on using Wikipedia as a credible academic information resource is still active (Tomaszewski and MacDonald, 2016).
Despite the efforts of Wikipedia’s contributors, many or even most articles in Wikipedia may still contain unsubstantiated or outdated claims, especially those flagged as being of lower quality (Lewoniewski et al., 2017). Sometimes editors might not use citations systematically (Chen and Roth, 2012; Forte et al., 2018) or engage in polarized edit conflicts (Umarova and Mustafaraj, 2019). Research suggests that some editors' violations might be caused by biases (Hube, 2017), such as cultural (Callahan and Herring, 2011), political (Greenstein and Zhu, 2012) or gender bias (Wagner et al., 2015).
Das and Lavoie (Das and Lavoie, 2014) determine the topics an editor is interested in and the editor’s stance by editors' behavior and interactions, finding that bias exists especially when a single point of view dominates controversial topics. Hube (2017) provides a method to detect both explicit and implicit bias in Wikipedia articles and observe its evolution by analyzing language, editing and citation styles. Greenstein and Zhu (2012) analyze political bias in Wikipedia by measuring the degree of political leaning of an article. They rely on a content-based method (Gentzkow and Shapiro, 2010) which calculates the frequency of particular phrases to measure the degree of political bias. They find that Wikipedia had a liberal bias in the early years, but that bias declines over time, supporting “a narrow interpretation of Linus' Law, namely, [that] a greater number of contributors to an article makes an article more neutral.” Besides, since Wikipedia has many language versions, different language versions can also contain specific biases. A study on the Wikipedia pages of UK politicians surfaced a substantial polarization of editors across political lines, in turn reflected in their choice of news media sources (Agarwal et al., 2020). Ewa and Susan (Callahan and Herring, 2011) find systematic biases in the focus on a particular topic or person in Wikipedia versions in different languages. Zhou et al. (2015) find that people’s attention to war-related topics affects the number of words and the number of subjective concepts, which in turn affects the bias of emotional expression. Last but not least, several studies have analyzed gender differences in Wikipedia. Wagner et al. (2015) find that while women on Wikipedia are covered and featured well in many Wikipedia language editions, the way women are portrayed might differ from the way men are portrayed. Reagle and Rhue (2011) study gender bias by comparing Wikipedia and Encyclopedia Britannica. They illustrate that while the number of articles related to women is increasing, compared to the articles on men, the articles on women are more likely to be missing on Wikipedia. Similarly, Zheng et al. investigated gender- and country-based biases in Wikipedia citation practices using data from the Web of Science and a Wikipedia citation dataset. Their findings showed that Wikipedia tends to cite publications by women less frequently than expected, and articles authored by women are less likely to be cited than those by men (Zheng et al., 2023). Researchers have also made efforts to improve the verifiability of Wikipedia’s contents for example by flagging unsupported contents in view of adding citations to reliable sources (Fetahu et al., 2016; Redi et al., 2019).
2.4 News media sources in Wikipedia
Wikipedia supports the use of sources from news media outlets: “news reporting from well-established news outlets is generally considered to be reliable for statements of fact” [7]. News media sources are indeed among the most-used in Wikipedia. Fetahu et al. (2015) find that almost 20% of the external references in the English version of Wikipedia are to news articles. In the dataset we use for this contribution such proportion is closer to 30%. Previous work also found that Wikipedia’s news sources are overall factually reliable, yet not uniformly so (Yang and Colavizza, 2022b). Nielsen uses a Wikipedia dump from 2008 to find that the BBC, the New York Times and the Washington Post were the most cited news media outlets at the time, with the BBC far ahead of the other outlets. Among the top 20 most-cited news outlets, most are American and four each being Australian and British (Nielsen, 2010). However, it is difficult to ignore the potential for polarization in news media (Patterson, 2011; Wolton, 2019), as well as their uneven reliability across the spectrum of outlets (Lazer et al., 2018). Previous work has focused on providing methods to assess the reliability of Wikipedia’s content. Hube and Fetahu (2018) proposed a supervised classification approach based on a self-build bias word lexicon, which could be able to detect biased statements with an accuracy of 74%. Przybyla et al. (2022) collect a corpus of over 50 million citations to 24 million identified sources from Wikipedia Complete Citation Corpus (WCCC) and build a search index using multiple meaning representations, using NLP (Natural Language Processing) and ML (Machine Learning), enabling the automatic retrieval of sources to support or disprove a claim. While a considerable amount of work has been done to assess the polarization and reliability of Wikipedia’s contents separately, their systematic and combined assessment for news media sources remains an open challenge.
3. Data
Our work is primarily based on Wikipedia Citations, a public dataset of citations from English Wikipedia to all its sources, including news media (Singh et al., 2021). We enrich Wikipedia Citations with data from the Media Bias Monitor (MBM) and the Media Bias Fact Check (MBFC): two authoritative indices of news media outlets providing an estimate of their political leaning and factual reliability. The combination of these sources allows us to quantify the political polarization and factual reliability of news sources cited from Wikipedia.
3.1 Wikipedia Citations
Wikipedia Citations includes more than 29M citations extracted from the over 6M articles English Wikipedia in May 2020. In Wikipedia Citations, each Wikipedia page contains several citations pointing to external sources. Of these, 25M (85.2%) are equipped with external links (URLs). For these URLs we extract the domain name using the tldextract package [8]. Domain names are critical in our approach as they allow linking to the Media Bias Monitor (MBM) and Media Bias Fact Check (MBFC) indices, which are based on Web domains and sub-domains. For example, given the external link https://www.nbcnews.com/politics/politics-news/al-gore-compares-climate-deniers-uvalde-law-enforcement-officers-nobod-rcna39707, after domain extraction we end up with www.nbcnews.com. With this method, we extract 1,554,632 unique domains. Then, we link domain names to MBM and MBFC by querying their APIs. The main limitation of this approach is that it works at the news media outlet level, which corresponds to the domain name, and not at the specific source level (the actual cited news article).
We further enrich Wikipedia Citations with information about an article’s topics and WikiProjects. With the data from ORES Web service [9] and public data (Johnson and Halfaker, 2020), we equip Wikipedia articles with topic (coverage of 99.3%) and WikiProjects (coverage of 17.5%). In this paper, we also use fractional counting to account for an article belonging to multiple topics or projects at the same time.
3.2 Media Bias Monitor (MBM)
To estimate the political polarization of Wikipedia citations, we use the Media Bias Monitor (Ribeiro et al., 2018). This system collects demographic data about the Facebook followers of 20,448 distinct news media outlets via Facebook Graph API [10] and Facebook Marketing API [11]. These data include political leanings, gender, age, income, ethnicity and national identity. For political leanings, the Facebook Audience API [12] provides five levels: Very Conservative, Conservative, Moderate, Liberal, Very Liberal. To measure the political leaning of an outlet, MBM firstly finds the fraction of readers having different political leanings, and then multiply the fraction for each category with the following values: very liberal (−2), liberal (−1), moderate (0), conservative (1) and very conservative (2). The sum of such scores provides a single polarization score for the outlet, ranging between −2 and 2, where a negative score indicates that a media outlet is read more by a liberal leaning audience, while a positive score indicates a conservative leaning audience. In the original paper, MBM is compared to alternative approaches used to infer the political leanings of news media outlets, finding that this method highly correlates with most alternatives. What is more, MBM covers 20,448 news media outlets and provides multi-dimension data. On these grounds, we use MBM in our study. As we have 1,554,632 unique domains from Wikipedia Citations, which mostly are not news media outlets, we focus the matching of unique domains to MBM on the most frequently cited domains from Wikipedia. In Figure A1, we show that the top 140,000 unique domains cover up to 90% of all Wikipedia Citations. We thus decided to only keep these top 140,000 domain names for our study, and match them in MBM.
When looking a domain name up via the MBM API [13], we can get four different query results:
No match; example: trove.nla.gov.au.
One exact match; example: www.breitbart.com.
More than one result, including the exact match; example: www.abc.com.
More than one result, not including an exact match; example: www.nytimes.com.
In each case, we proceed as follows. We label domains without a match (result 1) as NaN, while we use the polarization score of the exact match for domains with result 2 or 3. For domains with result 4, we use the average polarization score of all the matches, under the assumption that this approximates the polarization score of the exact match. To test our assumption, we use the 1,113 unique domains that have multiple results including an exact match (result 3), and compare the distribution of exact polarization scores and average polarization scores in Figure A2. We can see that the two distributions are overall comparable, which supports our assumption. Following this procedure, we are able to equip 4,866,377 citations (16.6% out of a total of 29.3 million) with polarization scores. These 4.9 million citations are all the citations in Figure A3b, while 29.3 million are 100% of citations in Figure A3a. We note again that 29.3 million is the total number of citations in the dataset, while citations to news media sources are estimated at 8.9 million (Yang and Colavizza (2022b)).
3.3 Media Bias Fact Check (MBFC)
To answer our research questions, we not only need the political polarization of a news media outlet, but also an estimate of its factual reliability. In order to get the reliability data, we use Media Bias Fact Check, which offers the largest set of labels of any news source rating service (Bozarth et al., 2020). For each news media outlet, a minimum of 10 headlines are reviewed and a minimum of five news stories are reviewed to get a reasonable factual rating [14]. MBFC classifies reliability in 6 levels: VERY HIGH (a score of 0), which means that the source is considered to be always factual; HIGH (a score of 1–2), which means that the source is considered to be almost always factual; MOSTLY FACTUAL (a score of 3–4), which means that the source is considered to be usually factual but may have failed a fact check or two that was not promptly corrected; MIXED (a score of 5–6), which means the source does not always use proper sourcing or sources to other mixed factual sources; LOW (a score of 7–9), which means the source rarely uses credible sources and is not trustworthy for reliable information; VERY LOW (a score of 10), which means the source rarely uses credible sources and is not trustworthy for reliable information. For example, in MBFC the New York Times is rated HIGH as they are mostly reliable except for some Op-Eds, and Fox News is rated MIXED because they may publish misleading reports.
We crawled all the news media outlet data on MBFC and got a dataset including the ratings for 3,586 outlets. Since we already have the domain names of each URLs in Wikipedia Citations, we use the same method to extract the domain names from the MBFC dataset as well. We then match the two datasets via domain names. 689 (19.2% out of 3,586) domains are matched resulting in 3,041,283 Wikipedia citations with both a factual rating and political polarization score, or 10.4% of all citations. In Figure 1 we show a bar plot of the number of Wikipedia citations by reliability scores, noting that, while there are only 1,467 citations rated as “VERY LOW,” there remains a sizable fraction of citations to low or mixed reliability outlets.
4. Results
We start by providing the overall distribution of Wikipedia’s citation political polarization score in Figure 2. We remind that the polarization score (x-axis) ranges between −2 (very liberal) and 2 (very conservative). The average Wikipedia citation polarization score (red line) is −0.51 (median −0.52), therefore leaning towards liberal. The bulk of citations also falls between the range −1 and 0.
We attempt to break down these results using information on Wikipedia’s articles, namely their topics and WikiProject. Topics are organized hierarchically, with four macro topics: culture, geography, history and society, STEM. The overview of citation political polarization per macro topic is given in Figure 3. On the left side, we use a violin plot to show the distribution of polarization scores for each macro topic. From this plot, we cannot see differences among macro topics. The bar plot on the right side provides the relative size of a topic in Wikipedia, showing how articles in Culture takes up nearly 50% of all citations, while STEM covers 6.5% of them.
Similarly, we show the distribution for the top 10 topics in Figure 4 and for the top 10 WikiProjects in Figure 5. We again confirm the general trend discussed above, while also finding minor shifts from it. For example, the topic sports has a higher conservative-leaning fraction of citations, all the while maintaining a liberal-leaning skew. The WikiProjects Politics and India are more liberal-leaning than the average, instead. Taken together, these results confirm that the overall trend toward liberal political polarization is not specific to some areas of Wikipedia but seems to be widespread across topics and WikiProjects.
In principle, in Wikipedia the neutrality and reliability of contents are tied together. Nevertheless, in practice, we speculate that editors may introduce political polarization in their sources in order to prioritize reliable ones. We have shown before, in Figure 1, that most cited news outlets are labeled as highly reliable or mostly factual, even if a significant share of mixed or low reliability sources remains. More details are given in Figure 6, where we plot the top 5 news outlets per polarization score group, and show their reliability class as well. In this plot, we divide the news outlets into four groups according to polarization scores: Very Liberal [−2 to −1], Liberal (−1 to 0], Conservative (0–1], Very Conservative (1–2]. For each group, the x-axis shows the percentage of a news outlet by citations within its group. We can see that, for example, The Guardian is labeled as mixed reliability and takes more than 50% of citations in the Very Liberal group, while the NYT is the second Very Liberal source and is considered highly reliable. Fox News is the top Very Conservative news outlet, with mixed reliability score. We note that in the group Liberal we also have YouTube, with a low reliability score. YouTube is not an outlet with an editorial policy per se, but a repository of contents of any kind. We therefore test whether our results hold when removing citations to YouTube from the dataset, finding that after removal the political polarization distribution moves slightly further liberal overall, while the effect of a conservative polarization in low reliability sources fades substantially. Nevertheless, these changes do not alter the main findings.
Next, we make use of multiple linear regression to address RQ2 and explore whether there is a relationship between political polarization and reliability of news media sources in Wikipedia. In our model, we take the political polarization score as the dependent variable, using Wikipedia article topics and WikiProjects as independent variables and set reliability as a control variable. To simplify the model, we proceed as follows. For topics, we use the macro topic: Geography, History and Society, Culture and STEM. For WikiProjects, we focus on the top 10 WikiProjects and set the rest of WikiProjects as “Other”. For reliability, we take MOSTLY FACTUAL as the reference class, which is the one in between high and low reliability classes. Thus, our final model is based on the following formula:
The results of our regression analysis are summarized in Table 1, providing a detailed examination of how political leaning is influenced by the interaction between news media reliability, Wikipedia topics, and WikiProjects. Although many of these effects are statistically significant, the coefficients demonstrate relatively modest magnitudes.
Directing our focus toward the impact of diverse Wikipedia topics and WikiProjects on political leaning, we discern that while a majority of Wikipedia topics indeed exert a significant influence on political leaning, their effects are generally limited in magnitude. Specifically, geography and STEM topics tend to incline toward conservative political leaning, whereas history and society topics demonstrate a propensity for liberal-leaning associations. In contrast, culture appears to exhibit a more encompassing character, potentially exerting minimal influence on political leaning. A similar scenario unfolds when considering the impact of WikiProjects, where history and society-affiliated projects, such as biography, tend to lean toward a liberal leaning, while STEM-related endeavors like medicine and pharmacology tend to align with a more conservative perspective. Intriguingly, projects tied to specific countries, such as the United States and India, manifest distinctive patterns. The US-associated project leans more toward conservatism, while the India-related project exhibits a predilection for liberalism. Overall, these findings align with our initial expectations and emphasize the complexity of determining political orientation within the Wikipedia framework.
As we delve into the assessment of news media reliability, the data does not unveil a straightforward or uniform pattern. Rather, it unveils a nuanced relationship between media reliability and political leaning. High reliability sources lean toward a liberal inclination, while very high reliability sources display a tendency toward conservatism. Conversely, mixed sources tend to favor a liberal perspective, while low and very low reliability sources align more closely with a conservative viewpoint.
To test our results, we develop several different models. First of all, we test only for polarization score and Wikipedia topics. In this model, all macro topics have a significant effect on the polarization score with small coefficients, that is, Geography and STEM will bring a less liberal skew while Culture History and Society will have an ever-stronger liberal skew. When using a model with reliability and topics, our results converge and become very similar to the model discussed above which also includes WikiProjects. As mentioned previously, we also test our final model without citations to YouTube. After removing them, the most important change is that the low reliability coefficient becomes non-significant and goes close to zero, thus making the case for a possible association between low reliability and conservative news outlets disappear.
5. Discussion
Wikipedia editors follow core policies when editing articles (Pavalanathan et al., 2018), in an attempt to provide a neutral point of view and reliable contents (Mesgari et al., 2015). Nevertheless, biases might still be found in Wikipedia in a variety of forms, and as such they require a never-ending effort on the part of the community. We find a moderate yet systematic liberal polarization in Wikipedia’s news media sources. The average polarization score of Wikipedia sources is −0.5, with the distribution of polarization scores concentrated around −1 to 0, on a scale between −2 (very liberal) and 2 (very conservative). These findings can be attributed to several factors, including the political leanings of Wikipedia contributors, the prominence and accessibility of liberal-oriented news sources, and potential methodological biases in gauging political polarization. Further investigation is required to gain a deeper understanding of the causative factors behind this observed polarization. Moreover, our results partially confirm and extend previous ones (Agarwal et al., 2020), while also showing that Wikipedia remains polarized towards liberal news media (Gentzkow and Shapiro, 2010). This finding is relevant as it signals a possible systematic biasing effect whose causes and effects will have to be further studied. News media sources not only select and provide specific information but also convey it with a certain framing which might influence how a topic is discussed in Wikipedia.
We initially speculated that the presence of political polarization might be partially explained by the editors' need to balance a source factual reliability with its political leaning. Interestingly, we find no clear relationship between reliability and polarization. The relationship between reliability and political polarization is complex, with more conservative sources being associated with both high and low reliability, while liberal sources tend to more often be of mixed reliability. This finding leaves the question of the motivations for political polarization open. We speculate that a multiplicity of factors might play a role, from pre-existing leanings in the composition of the editors' community, to an increasingly polarized media landscape making it difficult to find neutral news media sources to use. Our results may also help the case for changing Wikipedia’s sourcing policies, which might prevent information lacking accepted reliable secondary sources from being considered, and at the same time prevent broad and important areas of knowledge from entering the Wikipedia project. This is especially the case for cultures that rely on the non-written transmission and expression of knowledge (such as oral sources) (Taraborelli, 2019). Relying on a more diverse set of sources could be an approach to reduce possible bias in Wikipedia.
We acknowledge several limitations of our study, some of which constitute possible directions for future work. First of all, we rely also on external sources to measure political polarization and reliability. While such sources are considered authoritative, their coverage is only partial and they use specific approaches to score news media outlets which could be complemented in the future. We also mainly focus on the binary political polarization distinction between liberal and conservative sources. Several other political dimensions exist which could be considered in the future. Exploring the relationship between source polarization and reliability remains an open challenge, one that should focus on more dimensions than what we considered here, including time. In this respect, our work scores news media sources at the domain level, while a more granular analysis should be done at the level of the individual source contents. Similarly, we did not consider how a source is used in Wikipedia and whether its use reflects such polarization or not. Our study remains focused on English Wikipedia, while its extension to more languages would provide for a broader picture.
6. Conclusion
In this contribution, we analyzed a potential source of bias in Wikipedia, by considering citations to news media sources and their political polarization. We used a large-scale dataset of citations from Wikipedia, enriching it with metrics of political media polarization from the Media Bias Monitor, and of factual reliability from the Media Bias Fact Check. We found a moderate yet systematic liberal polarization in Wikipedia’s news media sources. We also showed that there is no clear relationship between a news media source’s reliability and its political leanings.
These results offer a foundation to inform Wikimedia’s research agenda about possible sources of disinformation and bias, in view of upholding its neutral point of view policy. Specifically, to keep Wikipedia as a neutral source of information, a better understanding of possible sources of bias is needed not only considering contents or editors, but also Wikipedia’s external sources. Here we provided a preliminary analysis of political polarization in Wikipedia from the perspective of citations to news media sources, while much works remains to be done. On the one hand, the measurement of political polarization, reliability and other potential signals of bias could be considered more comprehensively, although we have relied here on authoritative sources. On the other hand, a more granular study on the level of the contents of sources and how they are used in Wikipedia would significantly enrich our preliminary findings. The relationship between polarization and reliability would benefit from a higher dimensional analysis. Therefore, we see our work as fostering further attention to the always-open challenge of preserving and improving Wikipedia’s knowledge integrity.
Figures
Regression results for the effect of news media reliability on political leaning, controlling for Wikipedia topics and projects
| Variables | Coef | Std err | t | P>|t| | [0.025 | 0.975] |
|---|---|---|---|---|---|---|
| Intercept | −0.56 | 0.005 | −118.360 | 0.000 | −0.569 | −0.550 |
| C(factual, Treatment(reference = ’MOSTLY FACTUAL’))[T.VERY LOW] | 0.07 | 0.023 | 2.833 | 0.005 | 0.020 | 0.112 |
| C(factual, Treatment(reference = ’MOSTLY FACTUAL’))[T.LOW] | 0.09 | 0.003 | 30.143 | 0.000 | 0.083 | 0.095 |
| C(factual, Treatment(reference = ’MOSTLY FACTUAL’))[T.MIXED] | −0.20 | 0.002 | −82.082 | 0.000 | −0.201 | −0.192 |
| C(factual, Treatment(reference = ’MOSTLY FACTUAL’))[T.HIGH] | −0.09 | 0.002 | −37.653 | 0.000 | −0.090 | −0.081 |
| C(factual, Treatment(reference = ’MOSTLY FACTUAL’))[T.VERY HIGH] | 0.16 | 0.004 | 42.224 | 0.000 | 0.149 | 0.163 |
| Geography | 0.03 | 0.002 | 16.010 | 0.000 | 0.023 | 0.030 |
| Culture | −0.00 | 0.002 | −0.758 | 0.448 | −0.004 | 0.002 |
| History and Society | −0.02 | 0.001 | −16.579 | 0.000 | −0.027 | −0.021 |
| STEM | 0.01 | 0.002 | 7.098 | 0.000 | 0.009 | 0.016 |
| Biography | −0.04 | 0.002 | −23.958 | 0.000 | −0.047 | −0.039 |
| Medicine | 0.01 | 0.004 | 2.037 | 0.042 | 0.000 | 0.015 |
| Biography_science_and_academic_work_group | −0.04 | 0.004 | −10.502 | 0.000 | −0.047 | −0.032 |
| United_States | 0.06 | 0.002 | 36.096 | 0.000 | 0.060 | 0.067 |
| Articles_for_creation | 0.01 | 0.005 | 2.874 | 0.004 | 0.004 | 0.022 |
| Politics | −0.01 | 0.002 | −6.233 | 0.000 | −0.017 | −0.009 |
| India | −0.09 | 0.004 | −24.027 | 0.000 | −0.101 | −0.086 |
| Pharmacology | 0.11 | 0.007 | 14.672 | 0.000 | 0.092 | 0.120 |
| Lists | 0.04 | 0.003 | 15.955 | 0.000 | 0.037 | 0.047 |
| Military_history | 0.03 | 0.002 | 11.020 | 0.000 | 0.022 | 0.032 |
| Other | 0.02 | 0.004 | 3.986 | 0.000 | 0.008 | 0.023 |
Note(s): No. Observations: 604,459; R-squared: 0.047
Source(s): Created by author
Notes
References
Agarwal, P., Redi, M., Sastry, N., Wood, E. and Blick, A. (2020), “Wikipedia and westmin-ster: quality and dynamics of Wikipedia pages about UK politicians”, Proceedings of the 31st ACM Conference on Hypertext and Social Media, New York, NY, HT ’20, Association for Computing Machinery, pp. 161-166, doi: 10.1145/3372923.3404817.
Aragón, P. and Saez-Trumper, D. (2021), “A preliminary approach to knowledge integrity risk assessment in Wikipedia projects”, available at: https://arxiv.org/abs/2106.15940
Arazy, O., Morgan, W. and Patterson, R. (2006), “Wisdom of the crowds: decentralized knowledge construction in Wikipedia”, 16th Annual Workshop on Information Technologies and Systems (WITS) Paper, available at: https://papers.ssrn.com/abstract=1025624
Baigutanova, A., Myung, J., Saez-Trumper, D., Chou, A.-J., Redi, M., Jung, C. and Cha, M. (2023a), “Longitudinal assessment of reference quality on wikipedia”, Proceedings of the ACM Web Conference 2023, pp. 2831-2839, arXiv:2303.05227 [cs], available at: http://arxiv.org/abs/2303.05227
Baigutanova, A., Saez-Trumper, D., Redi, M., Cha, M. and Aragón, P. (2023b), “A com- parative study of reference reliability in multiple language editions of wikipedia”, arXiv:2309.00196 [cs], available at: http://arxiv.org/abs/2309.0019
Borra, E., Weltevrede, E., Ciuccarelli, P., Kaltenbrunner, A., Laniado, D., Magni, G., Mauri, M., Rogers, R. and Venturini, T. (2014), “Contropedia-the analysis and visualization of controversies in Wikipedia articles”, Proceedings of The International Symposium on Open Collaboration, pp. 1-1, doi: 10.1145/2641580.2641622.
Bozarth, L., Saraf, A. and Budak, C. (2020), “Higher ground? How ground truth labeling impacts our understanding of fake news about the 2016 us presidential nominees”, Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14, pp. 48-59, doi: 10.1609/icwsm.v14i1.7278, available at: https://ojs.aaai.org/index.php/ICWSM/article/view/7278
Callahan, E.S. and Herring, S.C. (2011), “Cultural bias in Wikipedia content on famous persons”, Journal of the American Society for Information Science and Technology, Vol. 62 No. 10, pp. 1899-1915, doi: 10.1002/asi.21577.
Chen, C.-C. and Roth, C. (2012), “{{Citation needed}} the dynamics of referencing in Wikipedia”, Proceedings of the eighth annual international symposium on wikis and open collaboration, pp. 1-4, doi: 10.1145/2462932.2462943.
Das, S. and Lavoie, A. (2014), “Automated inference of point of view from user interactions in collective intelligence venues”, International Conference on Machine Learning, PMLR, pp. 82-90, doi: 10.5555/3044805.3044902.
Fetahu, B., Anand, A. and Anand, A. (2015), “How much is Wikipedia lagging behind news?”, Proceedings of the ACM Web Science Conference, pp. 1-9.
Fetahu, B., Markert, K., Nejdl, W. and Anand, A. (2016), “Finding news citations for Wikipedia”, Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 337-346, doi: 10.1145/2983323.2983808.
Forte, A., Andalibi, N., Gorichanaz, T., Kim, M.C., Park, T. and Halfaker, A. (2018), “Information fortification: an online citation behavior”, Proceedings of the 2018 ACM Conference on Supporting Groupwork, pp. 83-92, doi: 10.1145/3148330.3148347.
Gentzkow, M. and Shapiro, J.M. (2010), “What drives media slant? Evidence from us daily newspapers”, Econometrica, Vol. 78 No. 1, pp. 35-71, doi: 10.3982/ECTA7195.
Greenstein, S. and Zhu, F. (2012), Collective Intelligence and Neutral Point of View: The Case of Wikipedia, technical report, National Bureau of Economic Research, available at: https://www.nber.org/papers/w18167
Hube, C. (2017), “Bias in wikipedia”, Proceedings of the 26th International Conference on World Wide Web Companion, pp. 717-721, doi: 10.1145/3041021.3053375.
Hube, C. and Fetahu, B. (2018), “Detecting biased statements in Wikipedia”, Companion proceedings of the web conference 2018, pp. 1779-1786, doi: 10.1145/3184558.3191640.
Johnson, I. and Halfaker, A. (2020), “Wikipedia articles and associated wikiproject templates”, available at: https://figshare.com/articles/dataset/Wikipedia_Articles_and_Associated_WikiProject_Templa
Kaffee, L.-A. and Elsahar, H. (2021), “References in Wikipedia: the editors' perspective”, Companion Proceedings of the Web Conference 2021, pp. 535-538, doi: 10.1145/3442442.3452337.
Lazer, D.M., Baum, M.A., Benkler, Y., Berinsky, A.J., Greenhill, K.M., Menczer, F., Metzger, M.J., Nyhan, B., Pennycook, G., Rothschild, D., Schudson, M., Sloman, S.A., Sunstein, C.R., Thorson, E.A., Watts, D.J. and Zittrain, J.L. (2018), “The science of fake news”, Science, Vol. 359 No. 6380, pp. 1094-1096, doi: 10.1126/science.aao2998, available at: https://www.science.org/doi/full/10.1126/science.aao2998
Lewoniewski, W., Wecel, K. and Abramowicz, W. (2017), “Relative quality and popularity evaluation of multilingual Wikipedia articles”, MDPI, Informatics, Vol. 4, p. 43, doi: 10.3390/informatics4040043, available at: https://www.mdpi.com/2227-9709/4/4/43
Lewoniewski, W., Wecel, K. and Abramowicz, W. (2019), “Multilingual ranking of Wikipedia articles with quality and popularity assessment in different topics”, Computers, Vol. 8 No. 3, p. 60, doi: 10.3390/computers8030060, available at: https://www.mdpi.com/2073-431X/8/3/60
Mesgari, M., Okoli, C., Mehdi, M., Nielsen, F.Å. and Lanamäki, A. (2015), “The sum of all human knowledge”: a systematic review of scholarly research on the content of Wikipedia”, Journal of the Association for Information Science and Technology, Vol. 66 No. 2, pp. 219-245, doi: 10.1002/asi.23172.
Messner, M. and South, J. (2011), “Legitimizing Wikipedia: how us national newspapers frame and use the online encyclopedia in their coverage”, Journalism Practice, Vol. 5 No. 2, pp. 145-160, doi: 10.1080/17512786.2010.506060.
Morgan, J. (2019), “Research: patrolling on Wikipedia”, available at: https://meta.wikimedia.org/wiki/Research:Patrolling_on_Wikipedia/Report
Nielsen (2010), “Top news cites referenced from Wikipedia”, available at: https://finnaarupnielsen.wordpress.com/2010/08/25/top-news-cites-referenced-from-wikipedia
Nielsen, F.Å., Mietchen, D. and Willighagen, E. (2017), “Scholia, scientometrics and wikidata”, European Semantic Web Conference, Springer, pp. 237-259, doi: 10.1007/978-3-319-70407-4_36.
Patterson, T.E. (2011), Out of Order: An Incisive and Boldly Original Critique of the News Media’s Domination of Ameri, A. Knopf, New York.
Pavalanathan, U., Han, X. and Eisenstein, J. (2018), “Mind your pov: convergence of articles and editors towards Wikipedia's neutrality norm”, Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW, pp. 1-23, doi: 10.1145/3274406.
Piccardi, T., Redi, M., Colavizza, G. and West, R. (2020), “Quantifying engagement with citations on Wikipedia”, Proceedings of The Web Conference 2020, pp. 2365-2376.
Przybyla, P., Borkowski, P. and Kaczyn´ski, K. (2022), “Countering disinformation by finding reliable sources: a citation-based approach”, 2022 International Joint Conference on Neural Networks (IJCNN), IEEE, pp. 1-8, ISSN: 2161-4407.
Reagle, J. and Rhue, L. (2011), “Gender bias in wikipedia and britannica”, International Journal of Communication, Vol. 5, p. 21, available at: http://hdl.handle.net/2047/d20002078
Redi, M., Fetahu, B., Morgan, J. and Taraborelli, D. (2019), “Citation needed: a taxonomy and algorithmic assessment of Wikipedia's verifiability”, The World Wide Web Conference, pp. 1567-1578, available at: doi: 10.1145/3308558.3313618.
Ribeiro, F.N., Henrique, L., Benevenuto, F., Chakraborty, A., Kulshrestha, J., Babaei, M. and Gummadi, K.P. (2018), “Media bias monitor: quantifying biases of social media news outlets at large-scale”, Twelfth international AAAI conference on web and social media, available at: https://ojs.aaai.org/index.php/ICWSM/article/view/15025
Rogers, R. and Sendijarevic, E. (2012), Neutral or National Point of View? a Comparison of Srebrenica Articles across Wikipedia’s Language Versions, Wikipedia Academy, Berlin.
Saez-Trumper, D. (2019), “Online disinformation and the role of Wikipedia”, available at: https://arxiv.org/abs/1910.12596
Sato, Y. (2021), “Non-English editions of Wikipedia have a misinformation problem”, available at: https://slate.com/technology/2021/03/japanese-wikipedia-misinformation-non-english-editions.html
Shi, F., Teplitskiy, M., Duede, E. and Evans, J.A. (2019), “The wisdom of polarized crowds”, Nature Human Behaviour, Vol. 3 No. 4, pp. 329-336, doi: 10.1038/s41562-019-0541-6, available at: http://arxiv.org/abs/1712.06414
Shuai, X., Jiang, Z., Liu, X. and Bollen, J. (2013), “A comparative study of academic and Wikipedia ranking”, Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries, pp. 25-28, doi: 10.1145/2467696.2467746.
Singh, H., West, R. and Colavizza, G. (2021), “Wikipedia citations: a comprehensive data set of citations with identifiers extracted from English Wikipedia”, Quantitative Science Studies, Vol. 2 No. 1, pp. 1-19, doi: 10.1162/qss_a_00105.
Smith, D.A. (2020), “Situating Wikipedia as a health information resource in various contexts: a scoping review”, PloS One, Vol. 15 No. 2, e0228786, doi: 10.1371/journal.pone.0228786.
Sugandhika, C. and Ahangama, S. (2022), “Assessing information quality of Wikipedia articles through Google's eat model”, IEEE Access, Vol. 10, pp. 52196-52209, doi: 10.1109/access.2022.3172962.
Sumi, R., Yasseri, T., Rung, A., Kornai, A. and Kertesz, J. (2011), “Edit wars in wikipedia”, in 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, IEEE, pp. 724-727, available at: http://arxiv.org/abs/1107.3689
Sutter, D. (2000), “Can the media be so liberal-the economics of media bias”, Cato Journal, Vol. 20, p. 431.
Taraborelli, D. (2019), “File: knowledge integrity - Wikimedia research 2030”, available at: https://meta.wikimedia.org/wiki/File:Knowledge_Integrity_Wikimedia_Research_2030.pdf
Thompson, N. and Hanley, D. (2018), “Science is shaped by Wikipedia: evidence from a randomized control trial”, MIT Sloan Research Paper, available at: https://www.ssrn.com/abstract=3039505
Tomaszewski, R. and MacDonald, K.I. (2016), “A study of citations to Wikipedia in scholarly publications”, Science and Technology Libraries, Vol. 35 No. 3, pp. 246-261, doi: 10.1080/0194262X.2016.1206052.
Umarova, K. and Mustafaraj, E. (2019), “How partisanship and perceived political bias affect Wikipedia entries of news sources”, Companion Proceedings of the 2019 World Wide Web Conference, New York, NY, WWW ’19, Association for Computing Machinery, pp. 1248-1253, doi: 10.1145/3308560.3316760.
Wagner, C., Garcia, D., Jadidi, M. and Strohmaier, M. (2015), “It's a man's Wikipedia? Assessing gender inequality in an online encyclopedia”, Proceedings of the international AAAI conference on web and social media, Vol. 9 No. 1, pp. 454-463, doi: 10.1609/icwsm.v9i1.14628, available at: http://arxiv.org/abs/1501.06307
Wikipedia (2022), “Vandalism on wikipedia”, available at: https://en.wikipedia.org/wiki/Vandalism_on_Wikipedia
Wolton, S. (2019), “Are biased media bad for democracy?”, American Journal of Political Science, Vol. 63 No. 3, pp. 548-562, doi: 10.1111/ajps.12424.
Yang, P. and Colavizza, G. (2022a), “A map of science in Wikipedia”, Proceedings of The Web Conference, available at: http://arxiv.org/abs/2110.13790
Yang, P. and Colavizza, G. (2022b), Science in Context: an Overview of All Wikipedia’s Sources, Technology and Innovation Indicators (STI 2022), Granada, doi: 10.5281/zenodo.6912664.
Yasseri, T., Spoerri, A., Graham, M. and Kertesz, J. (2013), “The most controversial topics in Wikipedia: a multilingual and geographical analysis”, SSRN Electronic Journal, doi: 10.2139/ssrn.2269392, available at: http://www.ssrn.com/abstract=2269392
Zheng, X., Chen, J., Yan, E. and Ni, C. (2023), “Gender and country biases in Wikipedia citations to scholarly publications”, Journal of the Association for Information Science and Technology, Vol. 74 No. 2, pp. 219-233, doi: 10.2139/ssrn.4360018, available at: https://onlinelibrary.wiley.com/doi/abs/10.1002
Zhou, Y., Cristea, A. and Roberts, Z. (2015), “Is Wikipedia really neutral? A sentiment perspective study of war-related Wikipedia articles since 1945”, Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation, pp. 160-168, available at: https://aclanthology.org/Y15-1019
Acknowledgements
Puyu Yang acknowledges the China Scholarship Council (CSC) grant.
Since submission of this article, the following author has updated their affiliation: Giovanni Colavizza is at the Department of Classical and Italian Philology, University of Bologna, Bologna, Italy.