Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10216

Avoid creating empty files during overwrite into Hive table with group by query

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.0.0
    • 2.3.0
    • SQL
    • None

    Description

      Exchange from GROUP BY query results in at least certain amount of partitions specified in 'spark.sql.shuffle.partition'.
      Hence, even when the number of distinct group-by key is small,
      INSERT INTO with GROUP BY query try to make at least 200 files (default value of 'spark.sql.shuffle.partition'),
      which results in lots of empty files.
      I think it is undesirable because upcoming queries on the resulting table will also make zero size partitions and unnecessary tasks do nothing on handling the queries.

      Attachments

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. SPARK-15393 Writing empty Dataframes doesn't save any _metadata files

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-15393 Writing empty Dataframes doesn't save any _metadata files

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21105 Useless empty files in hive table

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21435 Empty files should be skipped while write to file

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #8411 [Github] Pull Request #8411 (sirpkt)

          Pull request #12855 [Github] Pull Request #12855 (HyukjinKwon)

          Pull request #13181 [Github] Pull Request #13181 (marmbrus)

          Web Link [Github] Pull Request #18654 (xuanyuanking)

          Activity

            People

              gurwls223 Hyukjin Kwon
              sirpkt Keuntae Park
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: