When AI Moderation Fails Women, It Is a Governance Failure

We often hear that AI will make the internet safer. But what happens when these systems fail the very people they are supposed to protect?

When women face online abuse, the usual response is to improve the technology. Better data, better models, better detection. If platforms are not required to check how their systems affect women, technical improvements alone will not be enough.

A 2024 report of the UN Secretary-General cites estimates that 16–58 per cent of women and girls have experienced technology-facilitated violence. Meanwhile, Security Hero’s 2023 analysis estimated that 98 per cent of the deepfake videos it identified were pornographic, with women accounting for 99 per cent of individuals targeted in deepfake pornography.

How moderation fails women

A threat may be disguised as a joke. Misogyny may appear in coded language. A harassment campaign may involve hundreds of posts that seem harmless when viewed individually. Meanwhile, a woman describing the abuse she faced may find her own post removed.

A 2022 study by Kwarteng and colleagues found that widely used hate-speech detection tools, including Google’s Perspective API, were ineffective at detecting misogynoir, the intersection of racism and misogyny directed at Black women. The researchers also found that these systems could classify Black women’s self-advocacy as hateful, a problem they describe as computational tone policing.

Think about what that means. A system can miss abuse directed at a woman while incorrectly flagging her response to it. The problem is not simply whether the technology works. It is also whose experiences it recognises and whose voices it risks suppressing.

Incentives matter too. The Carnegie Endowment documents how women in politics are targeted by online abuse and gendered disinformation. Where outrage drives engagement, platforms may have little incentive to slow it down.

Commitments without sufficient accountability

The UN’s Global Digital Compact, adopted in September 2024, calls on technology companies to respect human rights and conduct human rights due diligence and impact assessments. But it is a political commitment, not a binding treaty.

The European Union goes further. Under Article 34 of the Digital Services Act, the largest platforms and search engines must assess systemic risks, including risks related to gender-based violence. Article 35 requires appropriate mitigation measures.

But a duty to assess is not a duty to look closely. A 2026 policy paper by bff, Germany’s federation of women’s counselling centres, found that gender-based violence was systematically downplayed in the platform assessments it reviewed.

India, where I am based, reflects a similar gap. Since February 2026, its IT Rules have required platforms to remove or disable access to intimate or morphed imagery within two hours of a complaint. Large platforms must also periodically review certain automated tools for accuracy, fairness, bias and discrimination. But the rules do not explicitly require these reviews to assess gendered harm or to be published.

This raises an uncomfortable question: if platforms assess their own risks, how can regulators and the public know whether those assessments are actually good enough?

What would change things

Platforms should be required to conduct dedicated gender impact assessments of automated moderation systems before deployment.

The European Institute for Gender Equality already provides a framework for examining how policies affect gender equality. Applying this approach to content moderation would mean testing systems against the forms of abuse women actually encounter, from coded misogyny and sexual deepfakes to coordinated harassment and threats in languages other than English.

Three things would make these assessments meaningful.

First, test before deployment. Platforms should identify and document foreseeable gender-related risks before launching a system or making significant changes to it.

Second, show the results. Reports should explain where systems fail, broken down by type of harm and language. They should measure both harmful content that slips through and legitimate speech that gets wrongly removed. Privacy must also be protected when working with sensitive data.

Third, allow independent scrutiny. Qualified auditors or regulators should be able to examine the methods, findings and corrective actions. Platforms should explain not only what went wrong but also what they did about it.

No moderation system will be perfect. But there is a difference between a system that makes mistakes and one whose failures are never properly examined.

The cost of waiting

Online abuse can push women away from public conversations, professional spaces and political participation. Moderation that misses the abuse, or silences the women who speak up, makes that worse.

We already have international commitments, legal frameworks and methods for assessing gender impact. What we need now is a clearer obligation to demonstrate that automated moderation systems take gendered harm seriously.

The question is not simply whether AI can moderate content at scale. It is whether platforms can prove that their systems protect women without silencing them.