HackerPapers

arXiv:2208.03274cs.CL

A Holistic Approach to Undesired Content Detection in the Real World

Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, Lilian Weng

Abstract

We present a holistic approach to building a robust and useful natural language classification system for real-world content moderation. The success of such a system relies on a chain of carefully designed and executed steps, including the design of content taxonomies and labeling instructions, data quality control, an active learning pipeline to capture rare events, and a variety of methods to make the model robust and to avoid overfitting. Our moderation system is trained to detect a broad set of categories of undesired content, including sexual content, hateful content, violence, self-harm, and harassment. This approach generalizes to a wide range of different content taxonomies and can be used to create high-quality content classifiers that outperform off-the-shelf models.

arXiv comments
Oral presentation at AAAI-23
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 2 identifiable commenters in 3 comments across 3 threads, from Jun 2023 to Jul 2023.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. Ask HN: Which recent research paper blew your mind? 1 comment

    • Content Moderation / Trust and Safety person Open AI’s como paper, A Holistic Approach to Undesired Content Detection in the Real World. https://arxiv.org/pdf/2208.03274.pdf Lots of interesting facts are strewn around the paper. ——- The first paper that squarely talked about the…

      intendedView on HNParent context

  2. Ask HN: How do you implement censorship to a LLM? 1 comment

    • I'm not sure what exactly you mean by censorship here because nothing that I know of that they do is actually censorship. In particular they don't do the sort of prompt stuffing you're talking about, because that would just use up the context window of the LLM and you'd know…

      seanhunterView on HNParent context

  3. The note Reddit sent to moderators threatening them if they don’t reopen 1 comment

    • Don't believe me? OpenAI themselves use a special-purpose model and not an LLM for moderation. https://arxiv.org/abs/2208.03274 If they don't think this is a situation for an LLM, you can probably safely assume it's not.

      seanhunterView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.