Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3334 Tez Custom Shuffle Handler
  3. TEZ-3363

Delete intermediate data at the vertex level for Shuffle Handler

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      For applications like pig where processing times can be very long, applications may choose to delete intermediate data for a sub dag. For example if a DAG has synced data to HDFS, all upstream intermediate data can be safely deleted.

      1. TEZ-3363.001.patch
        78 kB
        Kuhu Shukla
      2. TEZ-3363.002.patch
        78 kB
        Kuhu Shukla

        Activity

        Hide
        kshukla Kuhu Shukla added a comment -

        First cut of the patch.

        This patch adds a new event that is sent by every vertex to its ancestors at a given/configurable height. The ancestors are precomputed during DagPlan creation. When a vertex successfully completes, it sends this event to all the ancestors. A vertex that receives this event, calls vertexComplete() from the DagAppMaster all the way down to the ContainerLauncher and boiling down to the DeletionTracker. The vertex delete http calls to shuffle handler are sent serially right now to the nodes/shufflehandlers and that can be changed based on some feedback by sharing the threadpool or creating a new one. The shuffle handler does a regex match on the directory names and deletes the ones belonging to the passed vertex.
        In case a vertex fails vertex deletion does not quite kick in since the DAG will be failed and cleanup will follow shorty anyway.
        ShuffleUtils.isTezShuffleHandler calls have been added to VertexImpl which therefore has to import this new class, which may be undesired( ? ).
        Will let the precommit run and ask for some feedback soon.

        Show
        kshukla Kuhu Shukla added a comment - First cut of the patch. This patch adds a new event that is sent by every vertex to its ancestors at a given/configurable height. The ancestors are precomputed during DagPlan creation. When a vertex successfully completes, it sends this event to all the ancestors. A vertex that receives this event, calls vertexComplete() from the DagAppMaster all the way down to the ContainerLauncher and boiling down to the DeletionTracker. The vertex delete http calls to shuffle handler are sent serially right now to the nodes/shufflehandlers and that can be changed based on some feedback by sharing the threadpool or creating a new one. The shuffle handler does a regex match on the directory names and deletes the ones belonging to the passed vertex. In case a vertex fails vertex deletion does not quite kick in since the DAG will be failed and cleanup will follow shorty anyway. ShuffleUtils.isTezShuffleHandler calls have been added to VertexImpl which therefore has to import this new class, which may be undesired( ? ). Will let the precommit run and ask for some feedback soon.
        Hide
        tezqa TezQA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12886874/TEZ-3363.001.patch
        against master revision 7e895f5.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in :
        org.apache.tez.dag.app.launcher.TestContainerLauncherWrapper

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2628//testReport/
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2628//console

        This message is automatically generated.

        Show
        tezqa TezQA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12886874/TEZ-3363.001.patch against master revision 7e895f5. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in : org.apache.tez.dag.app.launcher.TestContainerLauncherWrapper Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2628//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2628//console This message is automatically generated.
        Hide
        tezqa TezQA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12886888/TEZ-3363.002.patch
        against master revision 7e895f5.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in :
        org.apache.tez.mapreduce.TestMRRJobsDAGApi

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2629//testReport/
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2629//console

        This message is automatically generated.

        Show
        tezqa TezQA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12886888/TEZ-3363.002.patch against master revision 7e895f5. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in : org.apache.tez.mapreduce.TestMRRJobsDAGApi Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2629//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2629//console This message is automatically generated.
        Hide
        kshukla Kuhu Shukla added a comment -

        This test TestMRRJobsDAGApi does not fail locally and seems like a case of TEZ-899.

        Show
        kshukla Kuhu Shukla added a comment - This test TestMRRJobsDAGApi does not fail locally and seems like a case of TEZ-899 .
        Hide
        kshukla Kuhu Shukla added a comment -

        Request for review/comments Jonathan Eagles/Siddharth Seth. Thanks a lot!

        Show
        kshukla Kuhu Shukla added a comment - Request for review/comments Jonathan Eagles / Siddharth Seth . Thanks a lot!

          People

          • Assignee:
            kshukla Kuhu Shukla
            Reporter:
            jeagles Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development