Big Data Systems (WT 2020/21)

Prof. Dr. Tilmann Rabl


The amount of data that can be generated and stored in academic and industrial projects and applications is increasing rapidly. Big data analytics technologies have established themselves as a solution for big data challenges to the scalability problems of traditional database systems. The vast amounts of new data that is collected, however, usually is not as easily analyzed as curated, structured data in a data warehouse is. Typically, these data are noisy, of varying format and velocity, and need to be analyzed with techniques from statistics and machine learning rather than pure SQL-like aggregations and drill-downs. Moreover, the results of the analyses frequently are models that are used for decision making and prediction. The complete process of big data analysis is described as a pipeline, which includes data recording, cleaning, integration, modeling, and interpretation.

In this lecture, we will discuss big data systems, i.e., infrastructures that are used to handle all steps in typical big data processing pipelines. We will learn about data center infrastructure and scale-out software systems. The software discussed will cover the full big data stack, i.e., distributed file systems, Map Reduce, key value stores, stream processing, graph processing, ML systems.

Predecessor of this series: Big Data Systems (WT 2019/20)

Introduction

Welcome

Date: October 2, 2020
Language: English
Duration: 00:02:09
Welcome 00:02:09

Introduction

Date: November 3, 2020
Language: English
Duration: 01:18:38

Big Data Stack

Introduction

Date: November 6, 2020
Language: English
Duration: 00:14:49

The Big Data Stack

Date: November 6, 2020
Language: German
Duration: 00:10:15

Google's Big Data Stack

Date: November 6, 2020
Language: German
Duration: 00:19:42

Open Source Big Data Stack

Date: November 6, 2020
Language: German
Duration: 00:15:59

Further Evolution

Date: November 6, 2020
Language: German
Duration: 00:07:22

MapReduce I

Introduction

Date: November 16, 2020
Language: English
Duration: 00:06:23

Map - Sort - Reduce

Date: November 16, 2020
Language: English
Duration: 00:14:47

Sorting in Detail

Date: November 16, 2020
Language: English
Duration: 00:16:25

MapReduce Architecture

Date: November 16, 2020
Language: English
Duration: 00:21:51

MR Algorithms

Date: November 16, 2020
Language: English
Duration: 00:11:26

Data Center and Cloud Computing

Introduction

Date: November 23, 2020
Language: English
Duration: 00:24:27

Virtualization

Date: November 23, 2020
Language: English
Duration: 00:16:29

Scheduling

Date: November 23, 2020
Language: English
Duration: 00:20:53

Cloud Computing

Date: November 23, 2020
Language: English
Duration: 00:14:12

Cloud Applications

Date: November 23, 2020
Language: English
Duration: 00:12:00

Distributed File Systems

Basics of File Systems

Date: November 25, 2020
Language: English
Duration: 00:22:37

Network File System

Date: November 25, 2020
Language: English
Duration: 00:16:52

Google File System

Date: November 25, 2020
Language: English
Duration: 00:20:13

Hadoop Distributed File System

Date: November 25, 2020
Language: English
Duration: 00:11:03

Erasure Coding

Date: November 25, 2020
Language: English
Duration: 00:09:46

(Big Data) File Format

Date: November 25, 2020
Language: English
Duration: 00:13:57

MapReduce II

MapReduce II

Date: November 30, 2020
Language: English
Duration: 00:13:12

Guest Lecture

Relational & Big-Data Processing in the Enterprise - Bridging the Gap

Date: January 6, 2021
Language: English
Duration: 00:57:17

Graph Databases

Graph Databases

Date: January 7, 2021
Language: English
Duration: 02:59:18

Machine Learning Systems

Introduction

Date: January 18, 2021
Language: German
Duration: 00:18:29

Machine Learning Models

Date: January 18, 2021
Language: German
Duration: 00:15:58

ML System Stack

Date: January 18, 2021
Language: German
Duration: 00:28:02

Language Abstraction & System Architectures

Date: January 18, 2021
Language: German
Duration: 00:18:46

SystemML

Date: January 18, 2021
Language: German
Duration: 00:28:52
SystemML 00:28:52

Execution Strategies

Date: January 18, 2021
Language: German
Duration: 00:44:08

Data-Parallel Parameter Server

Date: January 18, 2021
Language: German
Duration: 00:22:05

Federated Machine Learning

Date: January 18, 2021
Language: German
Duration: 00:13:48

Modern Hardware

Modern Hardware I

Date: January 26, 2021
Language: English
Duration: 00:19:31

Data Processing on GPUs

Date: January 26, 2021
Language: English
Duration: 00:37:26

Intro to Persistent Memory I

Date: January 26, 2021
Language: English
Duration: 00:10:23

Intro to Persistent Memory II

Date: January 26, 2021
Language: English
Duration: 00:20:25

A Brief Introduction to RDMAs

Date: January 26, 2021
Language: English
Duration: 00:22:00

Benchmarking & Measurement

Introduction

Date: February 1, 2021
Language: English
Duration: 00:08:49

Back of the Envelope Calculation

Date: February 1, 2021
Language: English
Duration: 00:12:06

Measurements & Metrics

Date: February 1, 2021
Language: English
Duration: 00:14:22

Some Statistics

Date: February 1, 2021
Language: English
Duration: 00:47:45

Benchmarks

Date: February 1, 2021
Language: English
Duration: 00:09:28

Sort Benchmarks

Date: February 1, 2021
Language: English
Duration: 00:16:25

BigBench / TPCx-BB - Big Data Benchmark

Date: February 1, 2021
Language: English
Duration: 00:13:48

Fair Benchmarking

Date: February 1, 2021
Language: English
Duration: 00:12:00

Guest Lecture