Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14269 Performance optimizations for data on S3
  3. HIVE-14272

ConditionalResolverMergeFiles should keep staging data on HDFS, then copy (no rename) to S3

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      If hive.merge.mapfiles is True, and the output table to write is on S3, then Hive will generate a conditional plan where smaller files will be merged into larger sizes.

      If the output files written by the initial MR job are small, then a second MR job is run to merge the output into larger files (a copy from S3 to S3 in the current code).

      If the original output files are large enough, then the conditional task is followed by a move/rename which is very expensive in S3.

      We should keep staging data on HDFS previous to copying them to S3 as final files.

        Attachments

          Activity

            People

            • Assignee:
              spena Sergio Peña
              Reporter:
              spena Sergio Peña
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: