Data Profiling and Data Cleansing (WS 2014/15)

Prof. Dr. Felix Naumann


Data profiling is the set of activities and processes to determine the metadata about a given dataset. Profiling data is an important and frequent activity of any IT professional and researcher.

It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the data set at hand. The first part of the lecture examines efficient detection methods for these properties.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing.

Many of the insights gained during data profiling point to deficiencies of the data. Profiling reveals data errors, such as inconsistent formatting within a column, missing values, or outliers. Profiling results can also be used to measure and monitor the general quality of a dataset, for instance by determining the number of records that do not conform to previously established constraints. The second part of the lecture examines various methods and algorithms to improve the quality of data, with an emphasis on the many existing duplicate detection approaches.

Introduction

Introduction

Date: October 13, 2014
Language: German
Duration: 01:29:33

An Introduction to Data Profiling

An Introduction to Data Profiling

Date: October 20, 2014
Language: German
Duration: 01:31:44

Visualization, Next Generation Profiling & Profiling Challenges

Date: October 23, 2014
Language: German
Duration: 01:24:32

Unique Column Combinations

Unique Column Combinations

Date: October 27, 2014
Language: German
Duration: 01:02:12

Detecting Inclusion Dependencies

Detecting Inclusion Dependencies

Date: November 10, 2014
Language: German
Duration: 01:20:03

SPIDER, Foreign Key Extraction & Conditional Inclusion Dependencies

Date: November 13, 2014
Language: German
Duration: 01:27:04

Der Apriori Algorithmus, Discovering cINDs & Detecting Functional Dependencies

Date: November 24, 2014
Language: German
Duration: 01:24:57

Detecting Functional Dependencies

TANE

Date: December 1, 2014
Language: German
Duration: 01:28:46

Dependency Checking, Approximate FDs, FD_Mine and DFD

Date: December 11, 2014
Language: German
Duration: 01:29:33

Conditional Uniques & IND Detection at Scale

Discovery of Conditional Unique Column Combination

Date: December 4, 2014
Language: German
Duration: 00:24:04

IND Detection on very many Tables

Date: December 4, 2014
Language: German
Duration: 00:41:02

Data Quality and Data Cleansing

Data Quality and Data Cleansing

Date: December 15, 2014
Language: German
Duration: 01:21:07

Duplicate Detection

Duplicate Detection

Date: December 18, 2014
Language: German
Duration: 01:30:07

Similarity Measures

Date: January 5, 2015
Language: German
Duration: 01:29:06

Similarity Measures & Generic Entity Resolution with Swoosh

Date: January 8, 2015
Language: German
Duration: 01:26:54

Sorted Neighborhood Methods

Date: January 15, 2015
Language: German
Duration: 01:25:58

Sorted Neighborhood Methods & Generic Entity Resolution with Swoosh

Date: January 19, 2015
Language: German
Duration: 01:25:48

Generic Entity Resolution with Swoosh

Date: January 26, 2015
Language: German
Duration: 00:44:04

Profiling Linked Data

Profiling Linked Data

Date: January 29, 2015
Language: English
Duration: 01:13:08