Big Data Systems (WT 2020/21)

Prof. Dr. Tilmann Rabl


The amount of data that can be generated and stored in academic and industrial projects and applications is increasing rapidly. Big data analytics technologies have established themselves as a solution for big data challenges to the scalability problems of traditional database systems. The vast amounts of new data that is collected, however, usually is not as easily analyzed as curated, structured data in a data warehouse is. Typically, these data are noisy, of varying format and velocity, and need to be analyzed with techniques from statistics and machine learning rather than pure SQL-like aggregations and drill-downs. Moreover, the results of the analyses frequently are models that are used for decision making and prediction. The complete process of big data analysis is described as a pipeline, which includes data recording, cleaning, integration, modeling, and interpretation.

In this lecture, we will discuss big data systems, i.e., infrastructures that are used to handle all steps in typical big data processing pipelines. We will learn about data center infrastructure and scale-out software systems. The software discussed will cover the full big data stack, i.e., distributed file systems, Map Reduce, key value stores, stream processing, graph processing, ML systems.

Predecessor of this series: Big Data Systems (WT 2019/20)

Introduction

Welcome

Date: October 2, 2020
Language: English
Duration: 00:02:09
Welcome 00:02:09

Introduction

Date: November 3, 2020
Language: English
Duration: 01:18:38

Big Data Stack

Introduction

Date: November 6, 2020
Language: English
Duration: 00:14:49

The Big Data Stack

Date: November 6, 2020
Language: German
Duration: 00:10:15

Google's Big Data Stack

Date: November 6, 2020
Language: German
Duration: 00:19:42

Open Source Big Data Stack

Date: November 6, 2020
Language: German
Duration: 00:15:59

Further Evolution

Date: November 6, 2020
Language: German
Duration: 00:07:22

MapReduce I

Introduction

Date: November 16, 2020
Language: English
Duration: 00:06:23

Map - Sort - Reduce

Date: November 16, 2020
Language: English
Duration: 00:14:47

Sorting in Detail

Date: November 16, 2020
Language: English
Duration: 00:16:25

MapReduce Architecture

Date: November 16, 2020
Language: English
Duration: 00:21:51

MR Algorithms

Date: November 16, 2020
Language: English
Duration: 00:11:26

Data Center and Cloud Computing

Introduction

Date: November 23, 2020
Language: English
Duration: 00:24:27

Virtualization

Date: November 23, 2020
Language: English
Duration: 00:16:29

Scheduling

Date: November 23, 2020
Language: English
Duration: 00:20:53

Cloud Computing

Date: November 23, 2020
Language: English
Duration: 00:14:12

Cloud Applications

Date: November 23, 2020
Language: English
Duration: 00:12:00

Distributed File Systems

Basics of File Systems

Date: November 25, 2020
Language: English
Duration: 00:22:37

Network File System

Date: November 25, 2020
Language: English
Duration: 00:16:52

Google File System

Date: November 25, 2020
Language: English
Duration: 00:20:13

Hadoop Distributed File System

Date: November 25, 2020
Language: English
Duration: 00:11:03

Erasure Coding

Date: November 25, 2020
Language: English
Duration: 00:09:46

(Big Data) File Format

Date: November 25, 2020
Language: English
Duration: 00:13:57

MapReduce II

MapReduce II

Date: November 30, 2020
Language: English
Duration: 00:13:12