Technical Report: Developing a Working Data Hub

Abstract

Data forms a key component of any enterprise. The need for high quality and easy access to data is further amplified by organizations wishing to leverage machine learning or artificial intelligence for their operations. To this end, many organizations are building resources for managing heterogenous data, providing end-users with an organization wide view of available data, and acting as a centralized repository for data owned/collected by an organization. Very broadly, we refer to these class of techniques as a "data hub." While there is no clear definition of what constitutes a data hub, some of the key characteristics include: data catalog; links to data sets or owners of data sets or centralized data repository; basic ability to serve / visualize data sets; access control policies that ensure secure data access and respects policies of data owners; and computing capabilities tied with data hub infrastructure. Of course, developing such a data hub entails numerous challenges. This document provides background in databases, data management and outlines best practices and recommendations for developing and deploying a working data hub.