Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14269 Performance optimizations for data on S3
  3. HIVE-15121

Last MR job in Hive should be able to write to a different scratch directory

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.2.0
    • Component/s: Hive
    • Labels:
      None

      Description

      Hive should be able to configure all intermediate MR jobs to write to HDFS, but the final MR job to write to S3.

      This will be useful for implementing parallel renames on S3. The idea is that for a multi-job query, all intermediate MR jobs write to HDFS, and then the final job writes to S3. Writing to HDFS should be faster than writing to S3, so it makes more sense to write intermediate data to HDFS.

      The advantage is that any copying of data that needs to be done from the scratch directory to the final table directory can be done server-side, within the blobstore. The MoveTask simply renames data from the scratch directory to the final table location, which should translate to a server-side COPY request. This way HiveServer2 doesn't have to actually copy any data, it just tells the blobstore to do all the work.

        Attachments

        1. HIVE-15121.3.patch
          29 kB
          Sahil Takiar
        2. HIVE-15121.2.patch
          29 kB
          Sahil Takiar
        3. HIVE-15121.1.patch
          21 kB
          Sahil Takiar
        4. HIVE-15121.patch
          7 kB
          Sahil Takiar
        5. HIVE-15121.WIP.2.patch
          10 kB
          Sahil Takiar
        6. HIVE-15121.WIP.1.patch
          10 kB
          Sahil Takiar
        7. HIVE-15121.WIP.patch
          10 kB
          Sahil Takiar

          Issue Links

            Activity

              People

              • Assignee:
                stakiar Sahil Takiar
                Reporter:
                stakiar Sahil Takiar
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: