Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5736

YARN container executor config does not handle white space

    Details

    • Hadoop Flags:
      Reviewed

      Description

      The container executor configuration reader does not handle white spaces or malformed key value pairs in the config file correctly or gracefully
      as an example the following key value line which is part of the configuration (note the << is used as a marker to show the extra trailing space):
      yarn.nodemanager.linux-container-executor.group=yarn <<
      is a valid line but when you run the check over the file:
      [root@test]#./container-executor --checksetup
      Can't get group information for yarn - Success.
      [root@test]#
      It fails to find the yarn group but it really tries to find the "yarn " group which fails. There is no trimming anywhere while processing the lines. If a space would be added in before or after the = sign a failure would also occur.
      Minor nit is the fact that a failure still is logged as a Success

      1. YARN_5736.000.patch
        4 kB
        Miklos Szegedi
      2. YARN-5736.001.patch
        4 kB
        Miklos Szegedi
      3. YARN-5736.002.patch
        8 kB
        Miklos Szegedi
      4. YARN-5736.addendum.000.patch
        0.7 kB
        Miklos Szegedi

        Activity

        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        I added a trim() function on both configuration keys and values. It removes all characters that satisfy isspace() from the beginning and the end of the strings. I also added error name 'unknown', if errno is not set to a valid error.

        Tested manually.

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - I added a trim() function on both configuration keys and values. It removes all characters that satisfy isspace() from the beginning and the end of the strings. I also added error name 'unknown', if errno is not set to a valid error. Tested manually.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 18s trunk passed
        +1 compile 0m 27s trunk passed
        +1 mvnsite 0m 26s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 23s the patch passed
        -1 cc 0m 23s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
        +1 javac 0m 23s the patch passed
        +1 mvnsite 0m 23s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 unit 14m 51s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        25m 23s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833407/YARN_5736.000.patch
        JIRA Issue YARN-5736
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux b154d0cdeadf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8a9f663
        Default Java 1.8.0_101
        cc https://builds.apache.org/job/PreCommit-YARN-Build/13397/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13397/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13397/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13397/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 18s trunk passed +1 compile 0m 27s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 13s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 23s the patch passed -1 cc 0m 23s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) +1 javac 0m 23s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 unit 14m 51s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 25m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833407/YARN_5736.000.patch JIRA Issue YARN-5736 Optional Tests asflicense compile cc mvnsite javac unit uname Linux b154d0cdeadf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8a9f663 Default Java 1.8.0_101 cc https://builds.apache.org/job/PreCommit-YARN-Build/13397/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13397/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13397/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13397/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Fixed Hadoop QA comments

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Fixed Hadoop QA comments
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 59s trunk passed
        +1 compile 0m 26s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 mvninstall 0m 21s the patch passed
        +1 compile 0m 24s the patch passed
        +1 cc 0m 24s the patch passed
        +1 javac 0m 24s the patch passed
        +1 mvnsite 0m 23s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 1s The patch has no whitespace issues.
        +1 unit 14m 56s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        25m 10s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833772/YARN-5736.001.patch
        JIRA Issue YARN-5736
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux 8d747c311f43 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ed9fcbe
        Default Java 1.8.0_101
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13411/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13411/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 59s trunk passed +1 compile 0m 26s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 24s the patch passed +1 cc 0m 24s the patch passed +1 javac 0m 24s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 unit 14m 56s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 25m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833772/YARN-5736.001.patch JIRA Issue YARN-5736 Optional Tests asflicense compile cc mvnsite javac unit uname Linux 8d747c311f43 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed9fcbe Default Java 1.8.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13411/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13411/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment -

        The patch looks good overall, but can you add a test for the trim function?

        Show
        rkanter Robert Kanter added a comment - The patch looks good overall, but can you add a test for the trim function?
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Adding unit test for trim

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Adding unit test for trim
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 13s trunk passed
        +1 compile 0m 27s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 mvninstall 0m 25s the patch passed
        +1 compile 0m 24s the patch passed
        +1 cc 0m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
        +1 javac 0m 24s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 unit 15m 51s hadoop-yarn-server-nodemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        26m 35s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e809691
        JIRA Issue YARN-5736
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837900/YARN-5736.002.patch
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux 155e8d0629cc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3fff158
        Default Java 1.8.0_101
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13817/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13817/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13817/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 13s trunk passed +1 compile 0m 27s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 14s trunk passed +1 mvninstall 0m 25s the patch passed +1 compile 0m 24s the patch passed +1 cc 0m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) +1 javac 0m 24s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 unit 15m 51s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 26m 35s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-5736 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837900/YARN-5736.002.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux 155e8d0629cc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fff158 Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-YARN-Build/13817/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13817/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13817/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        The unit test failure is unrelated. It appears to be YARN-5377

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - The unit test failure is unrelated. It appears to be YARN-5377
        Hide
        rkanter Robert Kanter added a comment -

        Thanks Miklos Szegedi. Committed to trunk!

        Show
        rkanter Robert Kanter added a comment - Thanks Miklos Szegedi . Committed to trunk!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10798 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10798/)
        YARN-5736. YARN container executor config does not handle white space (rkanter: rev 09f43fa9c089ebfc18401ce84755d3f2000ba033)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10798 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10798/ ) YARN-5736 . YARN container executor config does not handle white space (rkanter: rev 09f43fa9c089ebfc18401ce84755d3f2000ba033) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Parsing of container-executor.cfg appears to have been impacted by this patch. If container-executor.cfg contains entries with empty values, parsing doesn't work properly and all applications fail.

        Here is the c-e output with additional debugging.

        Failing this attempt.Diagnostics: Application application_1479220852002_0001 initialization failed (exitCode=255) with output: read_config :Conf file name is : /usr/local/src/hadoop_install/hadoop/etc/hadoop/container-executor.cfg
        read_config : Adding conf key : yarn.nodemanager.linux-container-executor.group
        read_config : Adding conf value : yarn
        read_config : Adding conf key : banned.users
        read_config : Adding conf key : min.user.id
        read_config : Adding conf value : 50
        read_config : Adding conf key : allowed.system.users
        read_config : Adding conf key : docker.binary
        read_config : Adding conf value : /usr/bin/docker
        read_config : Adding conf key : feature.docker.enabled
        read_config : Adding conf value : 1
        read_config : Adding conf key : feature.tc.enabled
        read_config : Adding conf value : 1
        Supplied key: yarn.nodemanager.linux-container-executor.group
        Compare key: yarn.nodemanager.linux-container-executor.group
        main : command provided 0
        main : run as user is nobody
        main : requested yarn user is root
        get_value Supplied key: min.user.id
        get_value Compare key: yarn.nodemanager.linux-container-executor.group
        get_value Compare key: min.user.ids
        get_value Compare key: docker.binarym.users
        get_value Compare key: feature.docker.enabled
        get_value Compare key: feature.tc.enabled
        Requested user nobody is not whitelisted and has id 99,which is below the minimum allowed 1000
        

        What was observed is that the min.user.id value is not properly parsed, because the key comparison in configuration.c#get_value will never succeed. This is because the keys have extra characters appended (min.user.id vs min.user.ids and docker.binary vs docker.binarym.users in the output above). Keys with an "empty" value (represented by # in container-executor.cfg) are being incorrectly combined with the key that follows in the config.

        The relevant min.user.id parts below:

        -snip-
        get_value Supplied key: min.user.id
        -snip-
        get_value Compare key: min.user.ids
        

        Note that banned.users and min.user.id are combined and docker.binary and allowed.system.users are combined.

        Here is an example container-executor.cfg that shows the issue:

        yarn.nodemanager.linux-container-executor.group=yarn
        banned.users=#
        min.user.id=50
        allowed.system.users=#
        docker.binary=/usr/bin/docker
        feature.docker.enabled=1
        feature.tc.enabled=1
        

        I haven't quite tracked down the root cause, but reverting this patch does appear to resolve the issue. I'm going to reopen this issue for further investigation.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Parsing of container-executor.cfg appears to have been impacted by this patch. If container-executor.cfg contains entries with empty values, parsing doesn't work properly and all applications fail. Here is the c-e output with additional debugging. Failing this attempt.Diagnostics: Application application_1479220852002_0001 initialization failed (exitCode=255) with output: read_config :Conf file name is : /usr/local/src/hadoop_install/hadoop/etc/hadoop/container-executor.cfg read_config : Adding conf key : yarn.nodemanager.linux-container-executor.group read_config : Adding conf value : yarn read_config : Adding conf key : banned.users read_config : Adding conf key : min.user.id read_config : Adding conf value : 50 read_config : Adding conf key : allowed.system.users read_config : Adding conf key : docker.binary read_config : Adding conf value : /usr/bin/docker read_config : Adding conf key : feature.docker.enabled read_config : Adding conf value : 1 read_config : Adding conf key : feature.tc.enabled read_config : Adding conf value : 1 Supplied key: yarn.nodemanager.linux-container-executor.group Compare key: yarn.nodemanager.linux-container-executor.group main : command provided 0 main : run as user is nobody main : requested yarn user is root get_value Supplied key: min.user.id get_value Compare key: yarn.nodemanager.linux-container-executor.group get_value Compare key: min.user.ids get_value Compare key: docker.binarym.users get_value Compare key: feature.docker.enabled get_value Compare key: feature.tc.enabled Requested user nobody is not whitelisted and has id 99,which is below the minimum allowed 1000 What was observed is that the min.user.id value is not properly parsed, because the key comparison in configuration.c#get_value will never succeed. This is because the keys have extra characters appended (min.user.id vs min.user.ids and docker.binary vs docker.binarym.users in the output above). Keys with an "empty" value (represented by # in container-executor.cfg) are being incorrectly combined with the key that follows in the config. The relevant min.user.id parts below: -snip- get_value Supplied key: min.user.id -snip- get_value Compare key: min.user.ids Note that banned.users and min.user.id are combined and docker.binary and allowed.system.users are combined. Here is an example container-executor.cfg that shows the issue: yarn.nodemanager.linux-container-executor.group=yarn banned.users=# min.user.id=50 allowed.system.users=# docker.binary=/usr/bin/docker feature.docker.enabled=1 feature.tc.enabled=1 I haven't quite tracked down the root cause, but reverting this patch does appear to resolve the issue. I'm going to reopen this issue for further investigation.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Adding addendum patch

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Adding addendum patch
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Thank you, Shane Kumpf, for reporting this! I attached a patch that should fix the issue based on your description. There was a bug in the code that it did not explicitly close the strings at the end, which strncpy does not do itself.

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you, Shane Kumpf , for reporting this! I attached a patch that should fix the issue based on your description. There was a bug in the code that it did not explicitly close the strings at the end, which strncpy does not do itself.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 44s trunk passed
        +1 compile 0m 28s trunk passed
        +1 mvnsite 0m 34s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 mvninstall 0m 27s the patch passed
        +1 compile 0m 29s the patch passed
        +1 cc 0m 29s the patch passed
        +1 javac 0m 29s the patch passed
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 unit 12m 54s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        25m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5736
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839027/YARN-5736.addendum.000.patch
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux ca5a3564f904 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5af572b
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13927/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13927/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 44s trunk passed +1 compile 0m 28s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 14s trunk passed +1 mvninstall 0m 27s the patch passed +1 compile 0m 29s the patch passed +1 cc 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 unit 12m 54s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 25m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5736 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839027/YARN-5736.addendum.000.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux ca5a3564f904 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5af572b Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13927/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13927/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks Miklos Szegedi for the quick turnaround! The addendum patch looks good to me and I can confirm that it fixes the problem.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks Miklos Szegedi for the quick turnaround! The addendum patch looks good to me and I can confirm that it fixes the problem.
        Hide
        templedf Daniel Templeton added a comment -

        +1 Checking in.

        Show
        templedf Daniel Templeton added a comment - +1 Checking in.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks for the quick fix, Miklos Szegedi! And that's for the validation, Shane Kumpf.

        Show
        templedf Daniel Templeton added a comment - Thanks for the quick fix, Miklos Szegedi ! And that's for the validation, Shane Kumpf .
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10841 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10841/)
        YARN-5736 Addendum. Fixes segfault due to unterminated string. (templedf: rev 264ddb13ff7455282fb640b6ff6c565adddea44e)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10841 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10841/ ) YARN-5736 Addendum. Fixes segfault due to unterminated string. (templedf: rev 264ddb13ff7455282fb640b6ff6c565adddea44e) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c

          People

          • Assignee:
            miklos.szegedi@cloudera.com Miklos Szegedi
            Reporter:
            miklos.szegedi@cloudera.com Miklos Szegedi
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development