Setting up SQL Server High-Availability between Windows and Linux with SQL Server 2017
Play Spark Performance Tuning - Part 4
Description
This week's Data Exposed show welcomes back Maxim Lukiyanov to talk more about Spark performance tuning with Spark 2.x. Maxim is a Senior PM on the big data HDInsight team and is in the studio today to present the final part of his 4-part series.
Topics in today's video:
[00:45] - Intro
[02:15] - Advanced Partitioning and Bucketing
[10:30] - Advanced Joins: Joining Large Tables
[19:00] - Debugging and Recap
Spark 2.2 rc4 on Azure HDInsight: Script action https://github.com/hdinsight/script-actions/tree/master/install-spark2-2
Download
Right click or Alt+Enter to download this episode
- MP3 (24.4 MB)
- Low Quality MP4 (51.1 MB)
- High Quality MP4 (854.6 MB)
- Mid Quality MP4 (177.1 MB)
The Discussion
-
runamuk00 The install link to Spark 2.2 rc4 on Azure HDInsight Github repo is wrong. The correct link is https://github.com/hdinsight/script-actions/tree/master/install-spark2-2
-
SQLScott @runamuk00:Thank you. The link is fixed.
-
Juanita Thank you for this great series ! Would be great to see how bucketing is coded in pyspark. Do you have some examples ?
-
maxluk @Juanita:Hi sure, Spark docs have good set of examples. Click on Python tab and you will see them in python. First example shows good basic use case, third example shows using partitioning and bucketing together.
More episodes in this series
Azure SQL Database Elastic Pools and Elastic Jobs
Related episodes
Spark Performance Tuning - Part 3
Spark Performance Tuning - Part 2
Spark Performance Tuning - Part 1
Create Spark Applications with the Azure Toolkit for IntelliJ
Optimizing HBase Performance in HDInsight
Packaging and deploying .NET Core for Linux - Part 1
Scaling .NET for Apache Spark processing jobs with Azure Synapse
Data processing with .NET for Apache Spark
SQL Server 2017 Adaptive QP
SQL Server 2017 Built-in Diagnostics
