Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3611 Support Docker Containers In LinuxContainerExecutor
  3. YARN-4004

container-executor should print output of docker logs if the docker container exits with non-0 exit status

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None

      Description

      When a docker container exits with a non-0 exit code, we should print the docker logs to make debugging easier.

      1. YARN-4004.003.patch
        5 kB
        Varun Vasudev
      2. YARN-4004.002.patch
        5 kB
        Varun Vasudev
      3. YARN-4004.001.patch
        5 kB
        Varun Vasudev

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/274/)
        YARN-4004. container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/274/ ) YARN-4004 . container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/266/)
        YARN-4004. container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/266/ ) YARN-4004 . container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2223 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2223/)
        YARN-4004. container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2223 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2223/ ) YARN-4004 . container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2204/)
        YARN-4004. container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2204/ ) YARN-4004 . container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1007 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1007/)
        YARN-4004. container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1007 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1007/ ) YARN-4004 . container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/277/)
        YARN-4004. container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/277/ ) YARN-4004 . container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8258 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8258/)
        YARN-4004. container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8258 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8258/ ) YARN-4004 . container-executor should print output of docker logs if the (xgong: rev c3364ca8e75acfb911ab92e19f357b132f128123) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt
        Hide
        xgong Xuan Gong added a comment -

        Committed into trunk/branch-2.
        Thanks, varun for the fix, and thanks, Sidharta Seethana for the review.

        Show
        xgong Xuan Gong added a comment - Committed into trunk/branch-2. Thanks, varun for the fix, and thanks, Sidharta Seethana for the review.
        Hide
        xgong Xuan Gong added a comment -

        +1 LGTM. Checking this in

        Show
        xgong Xuan Gong added a comment - +1 LGTM. Checking this in
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 6m 40s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 25s There were no new javac warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 yarn tests 6m 32s Tests passed in hadoop-yarn-server-nodemanager.
            24m 8s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12748563/YARN-4004.003.patch
        Optional Tests javac unit
        git revision trunk / 0306d90
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8760/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8760/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8760/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 6m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 25s There were no new javac warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 yarn tests 6m 32s Tests passed in hadoop-yarn-server-nodemanager.     24m 8s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748563/YARN-4004.003.patch Optional Tests javac unit git revision trunk / 0306d90 hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8760/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8760/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8760/console This message was automatically generated.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Xuan Gong , could you please review this patch?

        Thanks!

        Show
        sidharta-s Sidharta Seethana added a comment - Xuan Gong , could you please review this patch? Thanks!
        Hide
        sidharta-s Sidharta Seethana added a comment -

        +1 to the latest patch (003). Thanks, Varun Vasudev .

        Show
        sidharta-s Sidharta Seethana added a comment - +1 to the latest patch (003). Thanks, Varun Vasudev .
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 5m 32s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 51s There were no new javac warning messages.
        +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 21s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        -1 yarn tests 6m 6s Tests failed in hadoop-yarn-server-nodemanager.
            21m 45s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12748556/YARN-4004.002.patch
        Optional Tests javac unit
        git revision trunk / 0306d90
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8757/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8757/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8757/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 51s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. -1 yarn tests 6m 6s Tests failed in hadoop-yarn-server-nodemanager.     21m 45s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748556/YARN-4004.002.patch Optional Tests javac unit git revision trunk / 0306d90 hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8757/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8757/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8757/console This message was automatically generated.
        Hide
        vvasudev Varun Vasudev added a comment -

        Uploaded a new patch to address Sidharta Seethana's comments.

        Show
        vvasudev Varun Vasudev added a comment - Uploaded a new patch to address Sidharta Seethana 's comments.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Varun Vasudev ,

        I see that the read implementation for docker logs has been changed to read the entire output into a single buffer. While this is fine, the size of the buffer ( `PATH_MAX` ) could be problematic. Would it be better to use a different buffer with a different size (which is not tied to paths/PATH_MAX) for this purpose? The use of PATH_MAX here makes it a bit confusing.

          FILE* logs = popen(docker_logs_command, "r");
              if(logs != NULL) {
                clearerr(logs);
                res = fread(buffer, PATH_MAX, 1, logs);
                if(res < 1) {
                  fprintf(ERRORFILE, "%s %d %d\n",
                    "Unable to read from docker logs(ferror, feof):", ferror(logs), feof(logs));
                  fflush(ERRORFILE);
                }
                else {
                  fprintf(ERRORFILE, "%s\n", buffer);
                  fflush(ERRORFILE);
                }
              }
        
        Show
        sidharta-s Sidharta Seethana added a comment - Varun Vasudev , I see that the read implementation for docker logs has been changed to read the entire output into a single buffer. While this is fine, the size of the buffer ( `PATH_MAX` ) could be problematic. Would it be better to use a different buffer with a different size (which is not tied to paths/PATH_MAX) for this purpose? The use of PATH_MAX here makes it a bit confusing. FILE* logs = popen(docker_logs_command, "r" ); if (logs != NULL) { clearerr(logs); res = fread(buffer, PATH_MAX, 1, logs); if (res < 1) { fprintf(ERRORFILE, "%s %d %d\n" , "Unable to read from docker logs(ferror, feof):" , ferror(logs), feof(logs)); fflush(ERRORFILE); } else { fprintf(ERRORFILE, "%s\n" , buffer); fflush(ERRORFILE); } }
        Hide
        vvasudev Varun Vasudev added a comment -

        Thanks for the review Sidharta Seethana! Uploaded a new patch addressing your comments.

        Show
        vvasudev Varun Vasudev added a comment - Thanks for the review Sidharta Seethana ! Uploaded a new patch addressing your comments.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        For the segment of code below, would it be safer to print out the last few lines (few = some pre-determined constant) instead of the entire docker logs output? Given the duration of the time between 'docker run' and 'docker wait', this output is unlikely to be large, but just in case. Docker has a mechanism for this : docker logs --tail=<num> <container_id>

         snprintf(docker_logs_command, PATH_MAX, "%s logs %s",
                docker_binary, container_id);
              FILE* logs = popen(docker_logs_command, "r");
              while(fgets(buffer, PATH_MAX, logs) != NULL) {
                fprintf(ERRORFILE, "%s", buffer);
                fflush(ERRORFILE);
              }
        
        Show
        sidharta-s Sidharta Seethana added a comment - For the segment of code below, would it be safer to print out the last few lines (few = some pre-determined constant) instead of the entire docker logs output? Given the duration of the time between 'docker run' and 'docker wait', this output is unlikely to be large, but just in case. Docker has a mechanism for this : docker logs --tail=<num> <container_id> snprintf(docker_logs_command, PATH_MAX, "%s logs %s" , docker_binary, container_id); FILE* logs = popen(docker_logs_command, "r" ); while (fgets(buffer, PATH_MAX, logs) != NULL) { fprintf(ERRORFILE, "%s" , buffer); fflush(ERRORFILE); }
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 5m 18s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 35s There were no new javac warning messages.
        +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 20s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        -1 yarn tests 6m 3s Tests failed in hadoop-yarn-server-nodemanager.
            21m 10s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12748127/YARN-4004.001.patch
        Optional Tests javac unit
        git revision trunk / 93d50b7
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8733/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8733/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8733/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 18s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. -1 yarn tests 6m 3s Tests failed in hadoop-yarn-server-nodemanager.     21m 10s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748127/YARN-4004.001.patch Optional Tests javac unit git revision trunk / 93d50b7 hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8733/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8733/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8733/console This message was automatically generated.
        Hide
        vvasudev Varun Vasudev added a comment -

        Attached file with fix. The patch also fixes a bunch of compiler warnings and some unsafe code.

        Show
        vvasudev Varun Vasudev added a comment - Attached file with fix. The patch also fixes a bunch of compiler warnings and some unsafe code.

          People

          • Assignee:
            vvasudev Varun Vasudev
            Reporter:
            vvasudev Varun Vasudev
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development