Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      container-executor leaves behind docker command files that are used to run docker commands. These need to be cleaned up.

      1. YARN-4255.001.patch
        1 kB
        Sidharta Seethana

        Activity

        Hide
        sidharta-s Sidharta Seethana added a comment -

        Uploading trivial patch to clean up docker command files.

        Show
        sidharta-s Sidharta Seethana added a comment - Uploading trivial patch to clean up docker command files.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 5m 48s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 5s There were no new javac warning messages.
        -1 release audit 0m 18s The applied patch generated 1 release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 30s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 yarn tests 8m 47s Tests failed in hadoop-yarn-server-nodemanager.
            25m 4s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766110/YARN-4255.001.patch
        Optional Tests javac unit
        git revision trunk / e617cf6
        Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9409/artifact/patchprocess/patchReleaseAuditProblems.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9409/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9409/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9409/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 48s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 5s There were no new javac warning messages. -1 release audit 0m 18s The applied patch generated 1 release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 yarn tests 8m 47s Tests failed in hadoop-yarn-server-nodemanager.     25m 4s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766110/YARN-4255.001.patch Optional Tests javac unit git revision trunk / e617cf6 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9409/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9409/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9409/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9409/console This message was automatically generated.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Tests : The patch is trivial - manually tested with test applications that the command files are being deleted. YARN test failure is unrelated to this patch.

        Release audit check failure is unrelated to this patch.

        Show
        sidharta-s Sidharta Seethana added a comment - Tests : The patch is trivial - manually tested with test applications that the command files are being deleted. YARN test failure is unrelated to this patch. Release audit check failure is unrelated to this patch.
        Hide
        vvasudev Varun Vasudev added a comment -

        Sidharta Seethana - any reason this change is in container-executor and not in DockerRuntimeContainerExecutor? Also, moving this to be a sub-task of YARN-3611.

        Show
        vvasudev Varun Vasudev added a comment - Sidharta Seethana - any reason this change is in container-executor and not in DockerRuntimeContainerExecutor? Also, moving this to be a sub-task of YARN-3611 .
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Varun Vasudev , I am assuming you mean DockerLinuxContainerRuntime ? The reason for doing this in the container-executor is that the filename has a randomly generated component. We would have to save the filename in the NM state store (to persist across restarts) in order to clean this up effectively in java code. A similar pattern of cleaning up command files is already in place in container-executor (traffic shaping/tc command files are cleaned up in container-executor as well).

        Show
        sidharta-s Sidharta Seethana added a comment - Varun Vasudev , I am assuming you mean DockerLinuxContainerRuntime ? The reason for doing this in the container-executor is that the filename has a randomly generated component. We would have to save the filename in the NM state store (to persist across restarts) in order to clean this up effectively in java code. A similar pattern of cleaning up command files is already in place in container-executor (traffic shaping/tc command files are cleaned up in container-executor as well).
        Hide
        vvasudev Varun Vasudev added a comment -

        Makes sense. +1 to the patch. I'll commit tomorrow if no one objects.

        Show
        vvasudev Varun Vasudev added a comment - Makes sense. +1 to the patch. I'll commit tomorrow if no one objects.
        Hide
        vvasudev Varun Vasudev added a comment -

        Committed to trunk and branch-2. Thanks Sidharta Seethana!

        Show
        vvasudev Varun Vasudev added a comment - Committed to trunk and branch-2. Thanks Sidharta Seethana !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8630 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8630/)
        YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8630 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8630/ ) YARN-4255 . container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #527 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/527/)
        YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #527 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/527/ ) YARN-4255 . container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #539 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/539/)
        YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #539 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/539/ ) YARN-4255 . container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Thanks!

        Show
        sidharta-s Sidharta Seethana added a comment - Thanks!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2475 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2475/)
        YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2475 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2475/ ) YARN-4255 . container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1263 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1263/)
        YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1263 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1263/ ) YARN-4255 . container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #494 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/494/)
        YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #494 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/494/ ) YARN-4255 . container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2432 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2432/)
        YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2432 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2432/ ) YARN-4255 . container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c

          People

          • Assignee:
            sidharta-s Sidharta Seethana
            Reporter:
            sidharta-s Sidharta Seethana
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development