AI Content Moderation Tool for Social Media Giant

Client

A top global developer of social networking apps with dozens of millions of users

Industry

Social Media

Services

AI, ML, Data Science

Tech

AWS, CLIP, Python, microservices

Challenge

Live stream moderation on social media platforms is a challenging task, especially with the growing popularity of real-time broadcasts. While humans continue to be a crucial component of the moderation effort, technology is increasingly playing no less important role in scaling it up. In particular, a wealth of opportunities is presented by innovation in AI content moderation technologies. Adopting AI to protect online communities from abusive content was also a top priority for our long-standing client — one of the world’s largest developers of social networking apps that has to deal with millions of minutes of live streams broadcast on their apps daily. They understood that they needed to make the most of the latest tech advances to streamline content moderation processes. So when our dedicated data science team working on the client’s projects came up with an idea of how to leverage the power of AI to improve their processes, they were all hands up.

Our task was to:

●

Develop a vision for an AI-based content moderation tool with image classification capabilities powered by computer vision technology to identify live streams with abusive content

●

Identify methods that best match the problem, keeping scalability in mind

●

Create a dataset for AI model evaluation

●

Evaluate and fine-tune the AI model

●

Deploy the AI-based content moderation tool

Solution

We understood that we should develop an image classification solution that could be scaled with minimal effort to maximize cost savings for our client. We did R&D and figured out that OpenAI's new CLIP model would be a perfect fit. Trained using 400 million image-text pairs, i.e., a huge amount of labeled data, CLIP can understand the semantic meaning of images, providing a powerful bridge between computer vision and NLP. It is a zero-shot model, meaning that no retraining is needed to make the model perform the image classification tasks in domains it was not trained to do. In our project, we first focused on classification of Nazi-era symbol images as required by the client. So we created a dataset of thousands of relevant images, encoding both images and their describing texts to evaluate the CLIP model. Then we performed evaluation using metrics such as precision and recall and adjusted algorithm threshold values to fine-tune the model. The solution was implemented as a microservice using open-source libraries. To make the model classify images in new domains like weapons, drugs, etc., the client only needs to perform fine-tuning and update the configuration of the microservice.