Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39036

Support Alter Table/Partition Concatenate command

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • Spark Core, SQL
    • None

    Description

      Hi, folks, 

      In Hive, we can use following command to merge small files, however, there is not a corresponding command to do that in Spark SQL. 

      I believe it's useful and it's not enough only using AQE.  Is anyone working on this to merge small files? If not, I want to create a PR to implement it

       

      ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] CONCATENATE;

       

      https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionConcatenate

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            gabry.wu gabrywu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: