Description
How to reproduce:
spark-sql> set spark.sql.shuffle.partitions; spark.sql.shuffle.partitions 200 Time taken: 7.118 seconds, Fetched 1 row(s) spark-sql> set mapred.reduce.tasks=10; spark.sql.shuffle.partitions 10 Time taken: 5.968 seconds, Fetched 1 row(s) spark-sql> set mapreduce.job.reduces=20; set mapreduce.job.reduces 20 Time taken: 0.233 seconds, Fetched 1 row(s) spark-sql> set spark.sql.shuffle.partitions; spark.sql.shuffle.partitions 10 Time taken: 0.099 seconds, Fetched 1 row(s)
The SET mapreduce.job.reduces should automatically converted to spark.sql.shuffle.partitions, it's similar to set mapred.reduce.tasks.