Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23737

LLAP: Reuse dagDelete Feature Of Tez Custom Shuffle Handler Instead Of LLAP's dagDelete

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: llap

      Description

      LLAP have a dagDelete feature added as part of HIVE-9911, But now that Tez have added support for dagDelete in custom shuffle handler (TEZ-3362) we could re-use that feature in LLAP.
      There are some added advantages of using Tez's dagDelete feature rather than the current LLAP's dagDelete feature.

      1) We can easily extend this feature to accommodate the upcoming features such as vertex and failed task attempt shuffle data clean up. Refer TEZ-3363 and TEZ-4129

      2) It will be more easier to maintain this feature by separating it out from the Hive's code path.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                srahman Syed Shameerur Rahman
                Reporter:
                srahman Syed Shameerur Rahman
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m