Auto-Moderation of User Generated Content

Services: Artificial Intelligence / Machine Learning

Introduction

An e-commerce client from Europe approached us to tackle the increasing volume and evolving nature of toxicity, profanity, and spamming in their public Q&A forum and product reviews. They wanted to reduce the manual effort of their moderators who had to go through and report thousands of posts a day individually. They were also concerned about the threats of unethical hacking, indecent behavior, and data breaches due to Personal Identifiable Information (PII) exposure in their forum.

Our Approach

We developed NLP pipelines to clean, process, and extract relevant details from the web-scraped content. The user data and user-generated content were flagged by the client’s moderators, which helped us train and fine-tune machine learning models. We combined the client’s data with publicly available datasets of toxic posts and customized rule-based algorithms to detect PII and other blacklisted items. Dynamic training was used for enabling the models to automatically learn from newer posts and improve performance.

High level process flow diagram

Business Impact

Our solution achieved a prediction precision of 94% on the test data and post-production monitoring. We also achieved a true positive rate of 80% on the test data and post-production monitoring. Almost 75% of posts containing PII and other blacklisted items customized for the client were detected with almost full precision.
In addition, we migrated the data to a new database (PostgreSQL) beyond the agreement. This allowed the client to reduce hosting expenses and make adopting cloud services in the near future more feasible.

Conclusion

Our auto-moderation solution enabled our e-commerce client to manage the increasing volume and evolving nature of toxicity, profanity, and spamming in their user-generated content. It also reduced the manual effort of their moderators and improved data security by detecting PII and other blacklisted items. The successful implementation of this solution is a testament to our expertise in NLP pipelines, machine learning, and rule-based algorithms.