Community Notes Is Not The Silver Bullet Elon Musk Makes It Out To Be

As social media platforms grapple with the dual challenges of combating misinformation and preserving free speech, social platform X's 'community notes' programme — now adopted by Meta (United States) across its platforms — is emerging as a fascinating experiment in democratic content moderation.

This system, which empowers users to collaboratively fact-check and provide context to potentially misleading posts, represents a significant shift from traditional top-down moderation approaches.

An evolution that reflects a broader global dialogue: as platforms face mounting pressure to address disinformation, can community-driven moderation emerge as a potential solution for striking the delicate balance between protecting online free speech and curtailing harmful content?

The concept of community-driven moderation itself isn't groundbreaking — Wikipedia and Reddit have long relied on volunteer moderators. But X's approach charts new territory through its unprecedented transparency. With an open-source algorithm and a distinctive design that seeks to bridge ideological divides, community notes represents perhaps the most ambitious attempt yet to democratise content moderation. 

How Does it Work?

Community notes participation is open to X users who meet specific eligibility requirements. These include maintaining an active account for at least six months, having a clean record without recent rule infractions, and possessing a verified phone number.

Once enrolled, participants can contribute notes, which undergo peer assessment from fellow community note contributors.

The evaluation process involves other members rating each note on a three-tiered scale: they can mark it as fully helpful, partially helpful, or not helpful at all.

The platform employs a scoring mechanism where notes must achieve a rating exceeding 0.40 to become visible to the public. Notes falling below this threshold remain unpublished.

What makes this bridging algorithm different from a typical crowdsourcing one is its use of matrix factorisation.

Matrix factorisation makes use of linear regressions (a method to find the straight line that best predicts the relationship between two variables) to first identify the polarity of a user (example in terms of ideological inclinations — right, left, or centre), then extrapolate the data to find the polarity of each post.

A traditional crowdsourcing algorithm typically evaluates the quality of content by simply aggregating votes, such as averaging or summing them up.

For instance, a note with the most upvotes would generally be considered the best. However, the bridging algorithm takes a more sophisticated approach by focusing on the reasons behind voting behaviour rather than just the raw vote count.

It analyses the entire population of users and their voting patterns to uncover clusters of polarity. For example, if there is a group of Bharatiya Janata Party (BJP) supporters who frequently upvote fact-checks targeting pro-Congress accounts and vice versa, the algorithm identifies these voting clusters.

This allows it to predict that notes from specific users or on particular topics might attract votes due to partisan biases rather than the factual accuracy of their content.

By modelling these dynamics, the bridging algorithm ensures a more nuanced evaluation of quality that accounts for underlying voting tendencies.

The algorithm's uniqueness lies in its ability to acknowledge that while people may have political biases, they also possess other tendencies, such as upvoting content that is interesting, accurate, entertaining, or helpful, and downvoting content that is false or misleading.

The community notes algorithm deconstructs vote counts, distinguishing between votes influenced by users' perceptions of "helpfulness" and those driven by political biases.

This approach does not aim to give equal weight to opposing sides but rather to eliminate the impact of political bias, allowing for the extraction of more meaningful insights from users' votes.

By analysing voting behaviour and isolating the effects of bias, the algorithm reveals that most users are, to some extent, inclined toward valuing helpfulness.

This was demonstrated by a study conducted by Jonathan Warden (a philosophical software engineer), who plotted the community notes polarity two-dimensionally.

He observed:

“There are more users with a positive common-ground factor than a negative one. There is a clump of users in the upper-right quadrant because community notes users are overall right-leaning. But notice also that the helpfulness factor for these users is mostly above zero. They are also mostly biased towards helpfulness. These users are more likely to upvote posts that support a right-wing worldview, and also more likely to upvote posts that are helpful. This vertical component in these plots represents what I think of as common ground. It is something users tend to agree on independently of their politics.”

How well does this translate on the ground? Are community notes doing a better job than traditional fact-checkers? Can the effects of polarisation be effectively eliminated, and does that necessarily lead to greater quality, factually accurate, and objective dissemination of information?

Answers to these questions can best be found by using the litmus tests of the trinity — transparency, accountability, and reliability.

The algorithm associated with community notes, all the notes, ratings, and contributor data are publicly available and published daily. One can access the same and evaluate the mechanism used to arrive at a particular note in real time. A critical contrast to the vague and sometimes even opaque methodologies used by traditional fact-checking organisations. 

Apart from categorising the notes themselves, the two most important activities in community notes are writing and rating the notes. Contributors can see the history of their past 'writing impact' and 'rating impact' on their profiles.

Writing impact reflects how often a contributor's notes have earned the status of 'helpful' when rated by others. The impact increases when a note earns the 'helpful' status, and decreases when a note reaches the status of 'not helpful.'

Notes in the 'needs more ratings' status don't count towards writing impact. To increase one’s impact, a contributor needs to write notes that earn the status of 'helpful' when rated by other contributors.

These scores are shown publicly on contributor profiles, which means a contributor’s credentials are always out there, open to real-time scrutiny and change. This is different from the traditional systems, where reputation and trust often rely on static claims.

Here, it’s all about performance — how well someone’s contributions hold up — rather than just relying on a name or reputation, as often seen with traditional fact-checking platforms.

This is a more dynamic and merit-based approach.

Accountability 

While this addresses the accountability factor of contributors, what about the responsibility of the platforms themselves?

Critics raise valid questions about the true purpose of community notes. Some argue it's simply a clever way for platforms to dodge their content moderation responsibilities while cutting costs.

Instead of investing in robust fact-checking systems, are they simply offloading this crucial task to users under the guise of democratic participation?

How does this play out in the Indian context?

Section 79 of the Information Technology Act gives intermediaries immunity for third-party content, without limiting it to specific laws or regulations.

However, Section 79(3)(b) outlines conditions for losing this immunity, requiring intermediaries to promptly remove or disable access to unlawful material upon receiving actual knowledge. This raises the critical question of whether intermediaries must pre-screen all content they host.

However, in the Shreya Singhal case, the apex court interpreted that intermediaries need only act upon receiving a court order to remove or disable access to specific material and are thus not obligated to screen every upload.

Reliability

So far, so good. But not all is fine with the community notes in its present form. Like all technology, it has its limitations.

First is the issue of speed. While certainly faster than court orders or departmental instructions from the state, community notes are still too slow.

Most sharing of misinformation is in the immediate hours after it is posted. Community notes take time to be written and voted on, by which time (arguably) a lot of the damage has been done.

As noted by Tom Stafford (a Professor of Cognitive Science at the University of Sheffield), a report on misinformation about the Israel-Hamas conflict (that falsely attributed the image of Syrian victims of a chemical attack as Palestinians killed by Israelis) showed that relevant notes typically took more than seven hours to show up, with some taking even as long as 70 hours.

Additionally, making a cross-ideological agreement on truth the benchmark for the genesis of a community note may not always be the best approach for all forms of content.

For instance, as Warden points out in his blog, if the primary division within a user community isn't political but rather based on factors like expertise or knowledge level, the algorithm might mistakenly discount valuable insights from knowledgeable users.

For example, it could prioritise the opinions of less informed users simply because they align with a particular political viewpoint.

As the algorithm uses matrix factorisation to find a latent factor that best explains the variation amongst users’ votes, it assumes that this latent factor corresponds to some sort of polarisation within the community.

But what if the latent factor is due to diversity but not polarisation? What if the factor that best explains the variation in users’ votes corresponds to, for example, how informed or educated that user is?

A similar observation has also been made by Alex Mahadevan, the Poynter Institute’s MediaWise Director. Instead of aiding the combat of disinformation, sometimes the community notes themselves fail to recognise context and propagate misleading data, as was seen in this X post.

The community note essentially engages users in a data war, with both statistics, the one within the article and that within the note, focusing on entirely different subsets of people (the article speaks of present black fathers, while the note focusses on ‘single parent families’ and not exclusively black fathers).

The note only serves to take away from the human experiences the article seeks to highlight, with misleading and outdated data. It adds nothing of substance to the post and instead plays the role of a rabble-rouser.

Though the note was taken down after people downvoted it, it was still up for several days.

Another limitation is that community notes only apply to public posts. However, about 60 per cent of the most-rated notes are on private posts, which means the notes that are most needed on X posts aren't always visible to the public.

Still, research from X has shown that when community notes, especially those with cross-ideological agreement, are made public, they help slow the spread of misinformation.

According to X, people who saw these notes were significantly less likely to reshare the posts compared to those who didn’t see the annotations.

One study related to misinformation’s impact on public health, particularly concerning Covid-19, showed that community notes related to this topic were 97 per cent accurate.

Another study found that notes on misleading X posts reduced reposts by half and increased the chances that the original author would delete the X post by 80 per cent. This matches up with findings from X’s own research.

A particularly interesting finding came out of research conducted by Yang Gao, a professor at the Gies College of Business. He discovered that X users were more likely to delete their own posts after receiving community notes.

This approach is less likely to draw criticism for infringing on freedom of speech, as it prioritises user agency and encourages a more nuanced approach to addressing misinformation.

Thus, it balances between protecting First Amendment rights (Article 19 (2), or the fundamental right to speech, in the Indian context) and the urgent need to curb misinformation.

Gao also had an interesting suggestion. He recommended that X should notify not only the people who directly interacted with a post that received a note but also everyone who follows the person who wrote that post.

This is because his research showed that the 'observed influence' of a post — how widely it was shared and seen, regardless of the number of followers the author has — played a much bigger role in prompting users to retract their misinformation than presumed influence or simply how many followers they had.

Overall, while there are definitely some kinks to work out, like the speed of note creation and the potential for the algorithm to be misled by factors other than actual helpfulness, community notes has the potential to be a powerful tool.

It encourages users to take responsibility for the information they share and fosters a more nuanced approach to content moderation.

By combining community input with algorithmic analysis, we might be able to create a more informed and resilient online ecosystem.

It's definitely a work in progress, and an exciting development in the fight against misinformation.