Safety by design, online content moderation & community management

Blog Summary of the TFGBV Policy Dialogue Series – Session 5

One of the most debated questions—both in theory and in practice—is how much of technology-facilitated gender-based violence (TFGBV) can be traced back to the way platforms are designed: their architecture, safety thresholds, and the systems that often amplify harm instead of fostering community care. This is especially critical for populations already in situations of vulnerability. Increasingly, research points not only to the need to respond to digital violence, but to redesign the very systems that enable it in the first place.

The fifth session of the Technology-Facilitated Gender-Based Violence (TFGBV) Policy Dialogue Series, held on 16 October 2025, focused precisely on these questions under the theme “Safety by Design, Online Content Moderation & Community Management”. The session sought to identify key insights, practical mechanisms to ensure safety by design, and the main challenges ahead.

(Safety by design) ensures that users of our digital product, or any digital product, service, or platform, are protected from the very beginning, from the moment they sign up, (…), and ensure that they’re not being put further at risk simply by being a female user.” – Alexandra Tyers-Chowdhury

This session was part of the Policy Dialogue Series, an on-going initiative from UN Women and the SVRI Community of Practice on TFGBV to bring together governments, feminist civil society organisations, tech experts, human rights defenders, and researchers to address critical issues related to TFGBV.

Previous sessions in the Dialogue Series had explored the global and national challenges of addressing TFGBV, the realities faced by Women Human Rights Defenders (WHRDs) in digital spaces, and the growing influence of the manosphere as a space that reinforces online misogyny. These discussions provided a framework to understand TFGBV not as a series of isolated incidents, but as part of a structural ecosystem that reproduces and circulates violence.

Building on that foundation, this session asked a deeper question: what if the structure itself is complicit in spreading harm? Through this lens, the panel reflected on what lies beneath TFGBV and explored strategies that go beyond content removal approaches that encourage platforms to take proactive, design-level responsibility for prevention.

Moderated by Alexandra Tyers-Chowdhury, Chief Gender and Digital Officer at Panoply Digital, the session aimed to bridge research, policy, and practice to pinpoint priority areas for future work. Drawing on her experience designing digital products for women and girls, Alexandra framed the discussion through the lens of digital inclusion and safety for adolescent girls, emphasising how privacy, data protection, and trust are essential foundations of any online environment.

The imperative of “Safety by Design”

A central theme was the fundamental need to shift from reactive to proactive safety measures. In that way, the key question was, “How do we ensure we are not inadvertently causing more harm, and only relying on reactive measures after the fact, rather than proactive measures before the fact?”

This shift, from patching problems to redesigning systems, ran through the interventions of Sofia Bonilla and Gabriel Freeman from the Integrity Institute. They presented a two-pronged framework for proactive safety:

Behavior-Focused Interventions: These aim to influence user interactions in real-time to deter harmful behaviour. Examples include nudges, filters, and improved user onboarding processes.
Upstream Design Solutions: These proposals address the structural features of platforms, such as algorithmic systems, reporting methods, and rate limits, to tackle harm at its source.

Gabriel and Sofia discussed content moderation strategies, emphasising the need to move away from engagement-based content ranking to reduce the amplification of harmful content, including misogynistic and toxic narratives.

“One of the biggest issues social networks face is that people engage disproportionately with sensationalist and provocative content,” said Gabriel Freeman. “When content designed to provoke strong emotional reactions is rewarded by engagement algorithms, misogynistic content gains visibility and exacerbates TFGBV.”

They highlighted successful examples like YouTube’s transition to a combination of clicks and watch time, and Twitter’s use of community notes and raters to separate high ratings based on quality rather than partisanship (unfortunately the later is no longer in practice). Bonilla emphasized that user-controlled filters have become one of the most powerful design tools available for reclaiming online spaces.

“Filters are incredibly powerful tools that give users control over their online experiences. By enabling them to block specific keywords, topics, triggers, or emojis, users can effectively design their own digital boundaries. These tools don’t just hide harmful content, they let people define the limits of their online world, creating a protective buffer against TFGBV and other forms of harm.” – Sofia Bonilla

She cited Instagram’s Hidden Words, which reduced potentially offensive comments by 40%, and Faceit’s Minerva AI, both of which are now available to all users, regardless of follower count.

Proactive abuse prevention strategies

Expanding the discussion from moderation to technical design patterns, Denny George of Tattle Civic Technologies shared approaches that prioritise user agency. His team works on customizable filters and cross-platform signal-sharing systems that give individuals real control over their feeds. One of their most innovative projects crowdsourced slurs in Indian languages, building a localised dataset that empowers communities to flag and filter harmful content effectively.

Meanwhile, Angela Kanyi of THiNK, the Tech Innovators Network brought in her experience designing AI-driven chatbots. Her focus: embedding consent frameworks and safety dashboards into the development process from day one.

“Safety is integrated from the moment we start thinking about a project. At every sprint, we identify potential risks and mitigate them before they happen.” – Angela Kanyi

Gabriel Freeman added another dimension to this thinking, what he called continuous onboarding. Instead of frontloading all safety information before users join, platforms should reveal relevant features as users grow and their exposure changes.

“The new features are available to you, but also there’s new risks that you may want to adjust for. So in this new model of having this kind of continuous onboarding, it’s a lot easier to digest what the information is that you’re being provided.” – Gabriel Freeman

The youth perspective on reporting and support

From the policy world to the perspective of digital natives, Mona Cho, a high school student and member of Snapchat’s Council for Digital Wellbeing, offered a strikingly personal view. She described her evolution from seeing tech companies as “the enemy” to recognising that “to eradicate TFGBV requires a united front and deep collaboration.”

During her tenure, the council chose to focus on improving reporting systems. Though reporting is reactive rather than preventative, Mona argued that it remains a critical bridge for her generation, where stigma and silence still dominate.

She reflected on new updates to Snapchat’s interface that make reporting more friendly and specific, especially for sextortion cases.

“When you go to report now, it’s more friendly, and there’s a specific category for non-consensual image sharing. That felt like a big step forward.” — Mona Cho

She also mentioned Snapchat’s Keys to Digital Safety, an educational initiative co-created with civil society groups to help young users navigate online threats. But her most powerful point was about the offline dimension of harm.

Throughout the conversation, a recurring idea emerged. The online and offline worlds are deeply interconnected. As the panel discussed, sustainable safety requires cross-sector partnerships and human-centred systems that extend beyond the screen.

“Even if we have perfect reporting systems (..) TFGBV and the re-victimization it causes still persist in the hallways of classrooms. Local interventions and supporting the allies around young women, whether that be the young men in their lives, their caregivers, teachers, and school administrators are key to complementing and strengthening Safety by Design. Ultimately, that’s where impact truly happens: in schools, at home, in communities.” — Mona Cho

The Role of AI in content moderation

The conversation also turned to the complex and often contradictory role of artificial intelligence in shaping online safety. Moderator Alex Tyers-Chowdhury noted that while automation has become central to content moderation, it can also introduce new risks, especially for users who are already marginalised or underrepresented in training data.

Angela Kanyi, from THiNK, presented a detailed framework for building safe and ethical AI systems, grounded in three pillars: a Data Quality Framework that ensures ethically sourced, high-quality data; legal Compliance aligned with Kenya’s Data Protection Act, ISO standards, and a national AI Code of Practice; and a Conformity Assessment Process that rigorously evaluates AI tools before deployment.

Meanwhile, Denny George of Tattle Civic Technologies warned against framing AI as an “end-to-end solution” for identifying harmful or abusive content—a task that remains inherently subjective and context-dependent. Instead, he proposed using AI to assist human moderators, not replace them.

“AI is most effective when it supports the boring workflows—grouping variations of crowdsourced slurs, cleaning datasets, or flagging explicit material—so human moderators aren’t constantly exposed to traumatic content,” – Denny George.

He stressed that transparency and appeal mechanisms are essential in any system where machines make decisions about people’s speech or behaviour.

Co-creation of advocacy messages

The session closed with a collaborative reflection on how to translate these insights into action. Participants proposed measures such as age-responsive platform design, stronger data protection frameworks, and simplified, user-friendly reporting systems that reflect the diversity of global users.

To sustain this dialogue, participants were invited to contribute ideas through an open Miro board, where ongoing inputs will inform future advocacy messages and joint action plans. The aim is to ensure that the principles of Safety by Design—proactive prevention, accountability, and inclusion—move beyond discussion and begin to reshape digital policy and practice worldwide.