Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33828 SQL Adaptive Query Execution QA
  3. SPARK-35168

mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.2, 3.1.1, 3.2.0
    • 3.0.3, 3.1.2, 3.2.0
    • SQL
    • None

    Description

      spark-sql> set spark.sql.adaptive.coalescePartitions.initialPartitionNum=1;
      spark.sql.adaptive.coalescePartitions.initialPartitionNum	1
      Time taken: 2.18 seconds, Fetched 1 row(s)
      spark-sql> set mapred.reduce.tasks;
      21/04/21 14:27:11 WARN SetCommand: Property mapred.reduce.tasks is deprecated, showing spark.sql.shuffle.partitions instead.
      spark.sql.shuffle.partitions	1
      Time taken: 0.03 seconds, Fetched 1 row(s)
      spark-sql> set spark.sql.shuffle.partitions;
      spark.sql.shuffle.partitions	200
      Time taken: 0.024 seconds, Fetched 1 row(s)
      spark-sql> set mapred.reduce.tasks=2;
      21/04/21 14:31:52 WARN SetCommand: Property mapred.reduce.tasks is deprecated, automatically converted to spark.sql.shuffle.partitions instead.
      spark.sql.shuffle.partitions	2
      Time taken: 0.017 seconds, Fetched 1 row(s)
      spark-sql> set mapred.reduce.tasks;
      21/04/21 14:31:55 WARN SetCommand: Property mapred.reduce.tasks is deprecated, showing spark.sql.shuffle.partitions instead.
      spark.sql.shuffle.partitions	1
      Time taken: 0.017 seconds, Fetched 1 row(s)
      spark-sql>
      

      Attachments

        Activity

          People

            Qin Yao Kent Yao
            Qin Yao Kent Yao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: