Data quality challenges with missing values and mixed types in joint sequence analysis

Data quality challenges with missing values and mixed types in joint sequence analysis

TitleData quality challenges with missing values and mixed types in joint sequence analysis
Publication TypeConference Paper
Year of Publication2017
AuthorsAlina Lazar, Ling Jin, C. Anna Spurlock, Annika Todd, Kesheng Wu, Alex Sim
Conference Name2017 IEEE International Conference on Big Data (Big Data)2017 IEEE International Conference on Big Data (Big Data)
Date Published12/2017
PublisherIEEE
Conference LocationBoston, MA, USA
Abstract

The goal of this paper is to investigate the impact of missing values in categorical time series sequences on common data analysis tasks. Being able to more effectively identify patterns in socio-demographic longitudinal data is an important component in a number of social science settings. However, performing fundamental analytical operations, such as clustering for grouping these data based on similarity patterns, is challenging due to the categorical and multi-dimensional nature of the data, and their corruption by missing and inconsistent values. To study these data quality issues, we employ longitudinal sequence data representations, a similarity measure designed for categorical and longitudinal data, together with state-of-the art clustering methodologies reliant on hierarchical algorithms. The key to quantifying the similarity and difference among data records is a distance metric. Given the categorical nature of our data, we employ an “edit” type distance using Optimal Matching (OM). Because each data record has multiple variables of different types, we investigate the impact of mixing these variables in a single similarity measure. Between variables with binary values and those with multiple nominal values, we find that the ability to overcome missing data problems is harder in the nominal domain versus the binary domain. Additionally, artificial clusters introduced by the alignment of leading missing values can be resolved by tuning the missing value substitution cost parameter.

DOI10.1109/BigData.2017.8258222