Distributed Data Management (ST 2021) - tele-TASKhttps://www.tele-task.de/series/1349/The free lunch is over! Computer systems up until the turn of the century became constantly faster without any particular effort simply because the hardware they were running on increased its clock speed with every new release. This trend has changed and today's CPUs stall at around 3 GHz. The size of modern computer systems in terms of contained transistors (cores in CPUs/GPUs, CPUs/GPUs in compute nodes, compute nodes in clusters), however, still increases constantly. This caused a paradigm shift in writing software: instead of optimizing code for a single thread, applications now need to solve their given tasks in parallel in order to expect noticeable performance gains. Distributed computing, i.e., the distribution of work on (potentially) physically isolated compute nodes is the most extreme method of parallelization. Big data analytics and management are a multi-million dollar markets that grow constantly! The ability to control and utilize large amounts of data is the most valuable ability of today's computer systems. Because data volumes grow so rapidly and with them the complexity of questions they should answer, data analytics, i.e., the ability of extracting any kind of information from the data becomes increasingly difficult. As data analytics systems cannot hope for their hardware getting any faster to cope with performance problems, they need to embrace new software trends that let their performance scale with the still increasing number of processing elements. In this lecture, we take a look at various technologies involved in building distributed, data-intensive systems. We start by discussing fundamental concepts in distributed computing, such das data models, encoding formats, messaging, data replication and partitioning, fault tollerance, and batch- and stream processing. In between, we consider different practical systems from the Big Data Landscape, such as Akka and Spark. In the end, we concentrate on data management aspects, such as distributed database management system architectures and distributed query optimization.High quality e-learning content created with tele-TASK - more than video! Powered by Hasso Plattner Institute (HPI)Dr. Thorsten PapenbrockThe free lunch is over! Computer systems up until the turn of the century became constantly faster without any particular effort simply because the hardware they were running on increased its clock speed with every new release. This trend has changed and today's CPUs stall at around 3 GHz. The size of modern computer systems in terms of contained transistors (cores in CPUs/GPUs, CPUs/GPUs in compute nodes, compute nodes in clusters), however, still increases constantly. This caused a paradigm shift in writing software: instead of optimizing code for a single thread, applications now need to solve their given tasks in parallel in order to expect noticeable performance gains. Distributed computing, i.e., the distribution of work on (potentially) physically isolated compute nodes is the most extreme method of parallelization. Big data analytics and management are a multi-million dollar markets that grow constantly! The ability to control and utilize large amounts of data is the most valuable ability of today's computer systems. Because data volumes grow so rapidly and with them the complexity of questions they should answer, data analytics, i.e., the ability of extracting any kind of information from the data becomes increasingly difficult. As data analytics systems cannot hope for their hardware getting any faster to cope with performance problems, they need to embrace new software trends that let their performance scale with the still increasing number of processing elements. In this lecture, we take a look at various technologies involved in building distributed, data-intensive systems. We start by discussing fundamental concepts in distributed computing, such das data models, encoding formats, messaging, data replication and partitioning, fault tollerance, and batch- and stream processing. In between, we consider different practical systems from the Big Data Landscape, such as Akka and Spark. In the end, we concentrate on data management aspects, such as distributed database management system architectures and distributed query optimization.notele-TASKtele-task@hpi.deen℗; ©; tele-TASKTue, 03 Aug 2021 19:07:45 GMTPyRSS2Gen-1.1.0http://blogs.law.harvard.edu/tech/rssLecture Summaryhttps://www.tele-task.de/lecture/video/8704/Dr. Thorsten Papenbrock01:57:16tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8704/Wed, 21 Jul 2021 13:30:00 GMTFederated DBMSShttps://www.tele-task.de/lecture/video/8705/Dr. Thorsten Papenbrock01:16:07tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8705/Mon, 19 Jul 2021 11:00:00 GMTStream Processing - Databases and Streamshttps://www.tele-task.de/lecture/video/8692/Dr. Thorsten Papenbrock01:29:24tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8692/Mon, 12 Jul 2021 11:00:00 GMTStream Processinghttps://www.tele-task.de/lecture/video/8690/Dr. Thorsten Papenbrock01:20:16tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8690/Wed, 07 Jul 2021 13:30:00 GMTExercise 1 Evaluationhttps://www.tele-task.de/lecture/video/8689/Dr. Thorsten Papenbrock01:29:27tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8689/Mon, 05 Jul 2021 11:00:00 GMTSpark Batch Processing (2)https://www.tele-task.de/lecture/video/8673/Dr. Thorsten Papenbrock01:37:54tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8673/Wed, 30 Jun 2021 13:30:00 GMTSpark Batch Processinghttps://www.tele-task.de/lecture/video/8664/Dr. Thorsten Papenbrock01:29:30tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8664/Mon, 28 Jun 2021 11:00:00 GMTBatch Processing 2 - Distributed File Systems and MapReducehttps://www.tele-task.de/lecture/video/8652/Dr. Thorsten Papenbrock01:30:38tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8652/Mon, 21 Jun 2021 11:00:00 GMTBatch Processinghttps://www.tele-task.de/lecture/video/8648/Dr. Thorsten Papenbrock01:31:15tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8648/Wed, 16 Jun 2021 13:30:00 GMTTransactionshttps://www.tele-task.de/lecture/video/8643/Dr. Thorsten Papenbrock01:27:16tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8643/Mon, 14 Jun 2021 11:00:00 GMTConsistency and Consensushttps://www.tele-task.de/lecture/video/8639/Dr. Thorsten Papenbrock01:41:06tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8639/Wed, 09 Jun 2021 13:30:00 GMTDistributed Systemshttps://www.tele-task.de/lecture/video/8632/Dr. Thorsten Papenbrock01:21:07tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8632/Mon, 07 Jun 2021 11:00:00 GMTPartitioning & Distributed Systemshttps://www.tele-task.de/lecture/video/8629/Dr. Thorsten Papenbrock01:29:17tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8629/Wed, 02 Jun 2021 13:30:00 GMTReplication & Partitioninghttps://www.tele-task.de/lecture/video/8627/Dr. Thorsten Papenbrock01:22:22tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8627/Mon, 31 May 2021 11:00:00 GMTReplicationhttps://www.tele-task.de/lecture/video/8617/Dr. Thorsten Papenbrock01:35:19tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8617/Wed, 26 May 2021 21:15:00 GMTStorage and Retrievalhttps://www.tele-task.de/lecture/video/8614/Dr. Thorsten Papenbrock01:34:40tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8614/Wed, 19 May 2021 13:30:00 GMTData Models and Query Languageshttps://www.tele-task.de/lecture/video/8609/Dr. Thorsten Papenbrock01:34:25tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8609/Mon, 17 May 2021 11:00:00 GMTAkka Actor Programming 3 - Patternshttps://www.tele-task.de/lecture/video/8605/Dr. Thorsten Papenbrock00:42:13tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockThe first part of the lecture was lost due to technical problems. Here you find the complete lecture of the predecessor series: https://www.tele-task.de/lecture/video/7767/Dr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8605/Wed, 12 May 2021 13:30:00 GMTAkka Actor Programming 2https://www.tele-task.de/lecture/video/8599/Dr. Thorsten Papenbrock01:33:25tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8599/Wed, 05 May 2021 13:30:00 GMTAkka Actor Programminghttps://www.tele-task.de/lecture/video/8593/Dr. Thorsten Papenbrock01:32:32tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8593/Mon, 03 May 2021 11:00:00 GMTCommunication: Service-oriented & Database-oriented Middlewarehttps://www.tele-task.de/lecture/video/8585/Dr. Thorsten Papenbrock01:31:18tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8585/Wed, 28 Apr 2021 13:30:00 GMTCommunication: Message-oriented Middlewarehttps://www.tele-task.de/lecture/video/8579/Dr. Thorsten Papenbrock01:31:39tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8579/Mon, 26 Apr 2021 11:00:00 GMTCommunicationhttps://www.tele-task.de/lecture/video/8576/Dr. Thorsten Papenbrock01:23:58tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8576/Wed, 21 Apr 2021 13:30:00 GMTEncodinghttps://www.tele-task.de/lecture/video/8569/Dr. Thorsten Papenbrock01:33:49tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8569/Mon, 19 Apr 2021 11:00:00 GMTFoundationshttps://www.tele-task.de/lecture/video/8565/Dr. Thorsten Papenbrock01:22:24tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8565/Wed, 14 Apr 2021 13:30:00 GMTIntroductionhttps://www.tele-task.de/lecture/video/8561/Dr. Thorsten Papenbrock01:34:22tele-TASK, HPI, computer science, technology, Germany, PotsdamDr. Thorsten PapenbrockDr. Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/8561/Mon, 12 Apr 2021 11:00:00 GMT