Distributed Data Management (WT 2018/19)

Prof. Dr. Felix Naumann, Dr. Thorsten Papenbrock


The free lunch is over! Computer systems up until the turn of the century became constantly faster without any particular effort simply because the hardware they were running on increased its clock speed with every new release. This trend has changed and today's CPUs stall at around 3 GHz. The size of modern computer systems in terms of contained transistors (cores in CPUs/GPUs, CPUs/GPUs in compute nodes, compute nodes in clusters), however, still increases constantly. This caused a paradigm shift in writing software: instead of optimizing code for a single thread, applications now need to solve their given tasks in parallel in order to expect noticeable performance gains. Distributed computing, i.e., the distribution of work on (potentially) physically isolated compute nodes is the most extreme method of parallelization.

Big Data Analytics is a multi-million dollar market that grows constantly! Data and the ability to control and use it is the most valuable ability of today's computer systems. Because data volumes grow so rapidly and with them the complexity of questions they should answer, data analytics, i.e., the ability of extracting any kind of information from the data becomes increasingly difficult. As data analytics systems cannot hope for their hardware getting any faster to cope with performance problems, they need to embrace new software trends that let their performance scale with the still increasing number of processing elements.

In this lecture, we take a look a various technologies involved in building distributed, data-intensive systems. We discuss theoretical concepts (data models, encoding, replication, ...) as well as some of their practical implementations (Akka, MapReduce, Spark, ...). Since workload distribution is a concept which is useful for many applications, we focus in particular on data analytics.

Introduction

Introduction

Date: October 15, 2018
Language: English
Duration: 01:12:41

Lectures

Foundations

Date: October 16, 2018
Language: English
Duration: 01:23:06

Distributed DBMS

Date: October 22, 2018
Language: English
Duration: 01:29:01

Data Warehouses

Date: October 23, 2018
Language: English
Duration: 01:17:07

Encoding and Evolution

Date: October 29, 2018
Language: English
Duration: 01:11:08

Models of Dataflow

Date: October 30, 2018
Language: English
Duration: 01:27:18

Akka Actor-Programming Hands-on

Date: November 5, 2018
Language: English
Duration: 01:28:43

Akka Actor-Programming Part 2

Date: November 6, 2018
Language: English
Duration: 01:29:57

Patterns

Date: November 12, 2018
Language: German
Duration: 01:29:00
Patterns 01:29:00
Ask 00:19:51
Singleton 00:09:59
Reaper 00:19:28
Tutorial 00:39:42

Data Models and Query Languages

Date: November 13, 2018
Language: English
Duration: 01:24:05

Storage and Retrieval

Date: November 20, 2018
Language: German
Duration: 01:27:01

Replication

Date: November 26, 2018
Language: German
Duration: 01:26:56

Partitioning

Date: November 27, 2018
Language: German
Duration: 01:19:06

Batch Processing

Date: December 3, 2018
Language: English
Duration: 01:29:20

Distributed File Systems and MapReduce

Date: December 4, 2018
Language: English
Duration: 01:27:15

Beyond MapReduce

Date: December 10, 2018
Language: English
Duration: 01:29:41

Apache Spark

Date: December 11, 2018
Language: English
Duration: 01:29:38

Spark - Hands On

Date: December 17, 2018
Language: English
Duration: 01:28:41

Distributed Systems

Date: December 18, 2018
Language: English
Duration: 01:33:19

Consistency and Consensus

Date: January 7, 2019
Language: English
Duration: 01:30:02

Transactions

Date: January 8, 2019
Language: English
Duration: 01:29:57

Stream Processing

Date: January 14, 2019
Language: English
Duration: 01:27:31

Processing Streams

Date: January 15, 2019
Language: English
Duration: 01:33:58

Distributed Query Optimization (1)

Date: January 22, 2019
Language: English
Duration: 01:26:03

Distributed Query Optimization (2)

Date: January 22, 2019
Language: English
Duration: 01:23:23

Lecture Summary

Date: February 5, 2019
Language: English
Duration: 01:32:33

Apache Flink Training