Automated System for Improving RSS Feeds Data Quality

Abstract

Nowadays, the majority of RSS feeds provide incomplete information about their news items. The lack of information leads to engagement loss in users. We present a new automated system for improving the RSS feeds' data quality. RSS feeds provide a list of the latest news items ordered by date. Therefore, it makes it easy for a web crawler to precisely locate the item and extract its raw content. Then it identifies where the main content is located and extracts: main text corpus, relevant keywords, bigrams, best image and predicts the category of the item. The output of the system is an enhanced RSS feed. The proposed system showed an average item data quality improvement from 39.98% to 95.62%.