Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37217

The number of dynamic partitions should early check when writing to external tables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 3.2.0
    • 3.2.1, 3.3.0
    • SQL
    • None

    Description

      SPARK-29295 introduces a mechanism that writes to external tables is a dynamic partition method, and the data in the target partition will be deleted first.

      Assuming that 1001 partitions are written, the data of 10001 partitions will be deleted first, but because hive.exec.max.dynamic.partitions is 1000 by default, loadDynamicPartitions will fail at this time, but the data of 1001 partitions has been deleted.

      Attachments

        Activity

          People

            dzcxzl dzcxzl
            dzcxzl dzcxzl
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: