Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21279

Avoid moving/rename operation in FileSink op for SELECT queries

    XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      Currently at the end of a job FileSink operator moves/rename temp directory to another directory from which FetchTask fetches result. This is done to avoid fetching potential partial/invalid files by failed/runway tasks. This operation is expensive for cloud storage. It could be avoided if FetchTask is passed on set of files to read from instead of whole directory.

        Attachments

        1. HIVE-21279.1.patch
          31 kB
          Vineet Garg
        2. HIVE-21279.10.patch
          31 kB
          Vineet Garg
        3. HIVE-21279.11.patch
          33 kB
          Vineet Garg
        4. HIVE-21279.12.patch
          33 kB
          Vineet Garg
        5. HIVE-21279.13.patch
          33 kB
          Vineet Garg
        6. HIVE-21279.2.patch
          27 kB
          Vineet Garg
        7. HIVE-21279.3.patch
          27 kB
          Vineet Garg
        8. HIVE-21279.4.patch
          27 kB
          Vineet Garg
        9. HIVE-21279.5.patch
          27 kB
          Vineet Garg
        10. HIVE-21279.6.patch
          27 kB
          Vineet Garg
        11. HIVE-21279.7.patch
          27 kB
          Vineet Garg
        12. HIVE-21279.8.patch
          28 kB
          Vineet Garg
        13. HIVE-21279.9.patch
          30 kB
          Vineet Garg

          Issue Links

            Activity

              People

              • Assignee:
                vgarg Vineet Garg
                Reporter:
                vgarg Vineet Garg
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 50m
                  2h 50m