Distributed Data Management (WT 2018/19) - tele-TASKhttps://www.tele-task.de/series/1224/The free lunch is over! Computer systems up until the turn of the century became constantly faster without any particular effort simply because the hardware they were running on increased its clock speed with every new release. This trend has changed and today's CPUs stall at around 3 GHz. The size of modern computer systems in terms of contained transistors (cores in CPUs/GPUs, CPUs/GPUs in compute nodes, compute nodes in clusters), however, still increases constantly. This caused a paradigm shift in writing software: instead of optimizing code for a single thread, applications now need to solve their given tasks in parallel in order to expect noticeable performance gains. Distributed computing, i.e., the distribution of work on (potentially) physically isolated compute nodes is the most extreme method of parallelization. Big Data Analytics is a multi-million dollar market that grows constantly! Data and the ability to control and use it is the most valuable ability of today's computer systems. Because data volumes grow so rapidly and with them the complexity of questions they should answer, data analytics, i.e., the ability of extracting any kind of information from the data becomes increasingly difficult. As data analytics systems cannot hope for their hardware getting any faster to cope with performance problems, they need to embrace new software trends that let their performance scale with the still increasing number of processing elements. In this lecture, we take a look a various technologies involved in building distributed, data-intensive systems. We discuss theoretical concepts (data models, encoding, replication, ...) as well as some of their practical implementations (Akka, MapReduce, Spark, ...). Since workload distribution is a concept which is useful for many applications, we focus in particular on data analytics.High quality e-learning content created with tele-TASK - more than video! Powered by Hasso Plattner Institute (HPI)Prof. Dr. Felix Naumann, Thorsten PapenbrockThe free lunch is over! Computer systems up until the turn of the century became constantly faster without any particular effort simply because the hardware they were running on increased its clock speed with every new release. This trend has changed and today's CPUs stall at around 3 GHz. The size of modern computer systems in terms of contained transistors (cores in CPUs/GPUs, CPUs/GPUs in compute nodes, compute nodes in clusters), however, still increases constantly. This caused a paradigm shift in writing software: instead of optimizing code for a single thread, applications now need to solve their given tasks in parallel in order to expect noticeable performance gains. Distributed computing, i.e., the distribution of work on (potentially) physically isolated compute nodes is the most extreme method of parallelization. Big Data Analytics is a multi-million dollar market that grows constantly! Data and the ability to control and use it is the most valuable ability of today's computer systems. Because data volumes grow so rapidly and with them the complexity of questions they should answer, data analytics, i.e., the ability of extracting any kind of information from the data becomes increasingly difficult. As data analytics systems cannot hope for their hardware getting any faster to cope with performance problems, they need to embrace new software trends that let their performance scale with the still increasing number of processing elements. In this lecture, we take a look a various technologies involved in building distributed, data-intensive systems. We discuss theoretical concepts (data models, encoding, replication, ...) as well as some of their practical implementations (Akka, MapReduce, Spark, ...). Since workload distribution is a concept which is useful for many applications, we focus in particular on data analytics.notele-TASKtele-task@hpi.deen℗; ©; tele-TASKWed, 12 Dec 2018 17:05:37 GMTPyRSS2Gen-1.1.0http://blogs.law.harvard.edu/tech/rssApache Sparkhttps://www.tele-task.de/lecture/video/7228/Thorsten Papenbrock01:29:38tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7228/Tue, 11 Dec 2018 09:15:00 GMTBeyond MapReducehttps://www.tele-task.de/lecture/video/7226/Thorsten Papenbrock01:29:41tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7226/Mon, 10 Dec 2018 13:30:00 GMTDistributed File Systems and MapReducehttps://www.tele-task.de/lecture/video/7199/Thorsten Papenbrock01:27:15tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7199/Tue, 04 Dec 2018 09:15:00 GMTBatch Processinghttps://www.tele-task.de/lecture/video/7202/Thorsten Papenbrock01:29:20tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7202/Mon, 03 Dec 2018 13:30:00 GMTPartitioninghttps://www.tele-task.de/lecture/video/7180/Thorsten Papenbrock01:19:06tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7180/Tue, 27 Nov 2018 09:15:00 GMTReplicationhttps://www.tele-task.de/lecture/video/7179/Thorsten Papenbrock01:26:56tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7179/Mon, 26 Nov 2018 13:30:00 GMTStorage and Retrievalhttps://www.tele-task.de/lecture/video/7171/Thorsten Papenbrock01:27:01tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7171/Tue, 20 Nov 2018 09:15:00 GMTData Models and Query Languageshttps://www.tele-task.de/lecture/video/7131/Thorsten Papenbrock01:24:05tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7131/Tue, 13 Nov 2018 09:15:00 GMTPatternshttps://www.tele-task.de/lecture/video/7128/Thorsten Papenbrock01:29:00tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7128/Mon, 12 Nov 2018 13:30:00 GMTAkka Actor-Programming Part 2https://www.tele-task.de/lecture/video/7108/Thorsten Papenbrock01:29:57tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7108/Tue, 06 Nov 2018 09:15:00 GMTAkka Actor-Programming Hands-onhttps://www.tele-task.de/lecture/video/7107/Thorsten Papenbrock01:28:43tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7107/Mon, 05 Nov 2018 13:30:00 GMTModels of Dataflowhttps://www.tele-task.de/lecture/video/7097/Thorsten Papenbrock01:27:18tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7097/Tue, 30 Oct 2018 09:15:00 GMTEncoding and Evolutionhttps://www.tele-task.de/lecture/video/7096/Thorsten Papenbrock01:11:08tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7096/Mon, 29 Oct 2018 13:30:00 GMTData Warehouseshttps://www.tele-task.de/lecture/video/7079/Prof. Dr. Felix Naumann01:17:07tele-TASK, HPI, computer science, technology, Germany, PotsdamProf. Dr. Felix NaumannProf. Dr. Felix Naumannhttps://www.tele-task.de/lecture/video/7079/Tue, 23 Oct 2018 09:15:00 GMTDistributed DBMShttps://www.tele-task.de/lecture/video/7078/Prof. Dr. Felix Naumann01:29:01tele-TASK, HPI, computer science, technology, Germany, PotsdamProf. Dr. Felix NaumannProf. Dr. Felix Naumannhttps://www.tele-task.de/lecture/video/7078/Mon, 22 Oct 2018 13:30:00 GMTFoundationshttps://www.tele-task.de/lecture/video/7065/Thorsten Papenbrock01:23:06tele-TASK, HPI, computer science, technology, Germany, PotsdamThorsten PapenbrockThorsten Papenbrockhttps://www.tele-task.de/lecture/video/7065/Tue, 16 Oct 2018 09:15:00 GMTIntroductionhttps://www.tele-task.de/lecture/video/7064/Prof. Dr. Felix Naumann, Thorsten Papenbrock01:12:41tele-TASK, HPI, computer science, technology, Germany, PotsdamProf. Dr. Felix Naumann, Thorsten PapenbrockProf. Dr. Felix Naumann, Thorsten Papenbrockhttps://www.tele-task.de/lecture/video/7064/Mon, 15 Oct 2018 13:30:00 GMT