Is that a trick or a treat? Halloween changes the context for Trust and Safety teams

About a year ago, our team woke up to a sharp increase in content flagged for policy violations across several customer platforms. Our on-call engineers moved quickly to investigate, but the cause wasn't obvious. There had been no model updates or code changes in production. The client hadn't updated their policies. Where were all these violations coming from?

The culprit was Halloween. People were sharing photos of costumes, theatrical blood, gore makeup, and fake weapons. Our models interpreted some of those images as real graphic violence, leading to false positives.

The incident revealed a challenge for any Trust and Safety team using automation. Policy decisions depend on context, and context can shift with the calendar, culture, and current events. Our models had performed well on typical daily content and helped platforms make fast policy decisions without overenforcement. But our testing had not fully accounted for what might happen when the context changed.

Halloween is just one example of how context can shift quickly. Last week Meta's Oversight Board said Meta's automated content moderation incorrectly removed content linked to No Kings protests in the United States. The Board also criticized Meta for removing content related to Brazil's presidential election. In another case, posts about Breast Cancer Awareness Month were incorrectly removed because the model identified nudity in the informational images.

In the age of AI, an increasing level of detection work has shifted from human moderators to AI and machine learning models. Effective detection work takes context into consideration, and we must enable our models to accommodate changing landscapes. In Trust and Safety, we often focus on catching new threats, especially ones designed to evade detection systems. But it is also important to ensure we do not overenforce policies. Last year, changing context led to some Halloween costumes being temporarily removed. But in other cases, overenforcement could lead to automated decisions that influence elections or impact healthcare. In the past year, we've taken steps to make our systems more resilient:

• We've continued improving our models so they can better account for cultural and temporal context. They now adapt as content patterns shift.
• We've strengthened monitoring to help us spot changes in content and model behavior earlier.
• We've added proactive testing around predictable events. For example, before Halloween, we tested models on a curated set of Halloween makeup and costume images to check whether they could distinguish staged effects from real-world violence.

No detection program can anticipate every shift in content. New trends, threats, and forms of expression emerge as society and technology change. That's why Trust and Safety systems require ongoing attention. These models must help platforms protect people from harm while recognizing that context and behavior evolve. At Cinder, we're committed to continuously improving harm detection so it remains accurate, useful, and responsive to those changes.