Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33828 SQL Adaptive Query Execution QA
  3. SPARK-35168

mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.2, 3.1.1, 3.2.0
    • Fix Version/s: 3.0.3, 3.1.2, 3.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      spark-sql> set spark.sql.adaptive.coalescePartitions.initialPartitionNum=1;
      spark.sql.adaptive.coalescePartitions.initialPartitionNum	1
      Time taken: 2.18 seconds, Fetched 1 row(s)
      spark-sql> set mapred.reduce.tasks;
      21/04/21 14:27:11 WARN SetCommand: Property mapred.reduce.tasks is deprecated, showing spark.sql.shuffle.partitions instead.
      spark.sql.shuffle.partitions	1
      Time taken: 0.03 seconds, Fetched 1 row(s)
      spark-sql> set spark.sql.shuffle.partitions;
      spark.sql.shuffle.partitions	200
      Time taken: 0.024 seconds, Fetched 1 row(s)
      spark-sql> set mapred.reduce.tasks=2;
      21/04/21 14:31:52 WARN SetCommand: Property mapred.reduce.tasks is deprecated, automatically converted to spark.sql.shuffle.partitions instead.
      spark.sql.shuffle.partitions	2
      Time taken: 0.017 seconds, Fetched 1 row(s)
      spark-sql> set mapred.reduce.tasks;
      21/04/21 14:31:55 WARN SetCommand: Property mapred.reduce.tasks is deprecated, showing spark.sql.shuffle.partitions instead.
      spark.sql.shuffle.partitions	1
      Time taken: 0.017 seconds, Fetched 1 row(s)
      spark-sql>
      

        Attachments

          Activity

            People

            • Assignee:
              Qin Yao Kent Yao
              Reporter:
              Qin Yao Kent Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: