Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1519

check if sysconf is implemented before using it

    Details

      Description

      If sysconf value _SC_GETPW_R_SIZE_MAX is not implemented, it leads to segfault because invalid pointer gets passed to libc function.

      fix: enforce minimum value 1024, same method is used in hadoop-common native code.

      1. nodemgr-sysconf.txt
        2 kB
        Radim Kolar
      2. YARN-1519.002.patch
        2 kB
        Eric Payne
      3. YARN-1519.003.patch
        2 kB
        Eric Payne

        Activity

        Hide
        hsn Radim Kolar added a comment -

        fixed reported case and also removed compiler warning in while loop.

        Show
        hsn Radim Kolar added a comment - fixed reported case and also removed compiler warning in while loop.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12619450/nodemgr-sysconf.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2698//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2698//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619450/nodemgr-sysconf.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2698//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2698//console This message is automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        Radim Kolar, thank you for raising this issue and submitting the patch.

        With the recent changes to the Apache build, I'm pretty sure that patches should take the form of YARN-1519.002.patch, especially the .patch so that it will kick the build.

        I have taken the liberty of upmerging the patch and running test-patch.sh against it. All is fine except that there are no unit tests. In order to test this patch effectively, one would have to cause sysconf(_SC_GETPW_R_SIZE_MAX) to return -1, and I can't think of a way to do that without creating a custom sysconf() object, creating its own share library, and putting that in LD_LIBRARY_PATH ahead of libc. Can you think of a better way to test this? Otherwise, I would suggest to leave the patch as it is.

        +1, by the way (non-binding)

        Show
        eepayne Eric Payne added a comment - Radim Kolar , thank you for raising this issue and submitting the patch. With the recent changes to the Apache build, I'm pretty sure that patches should take the form of YARN-1519 .002.patch , especially the .patch so that it will kick the build. I have taken the liberty of upmerging the patch and running test-patch.sh against it. All is fine except that there are no unit tests. In order to test this patch effectively, one would have to cause sysconf(_SC_GETPW_R_SIZE_MAX) to return -1, and I can't think of a way to do that without creating a custom sysconf() object, creating its own share library, and putting that in LD_LIBRARY_PATH ahead of libc . Can you think of a better way to test this? Otherwise, I would suggest to leave the patch as it is. +1, by the way (non-binding)
        Hide
        raviprak Ravi Prakash added a comment -

        Nitpick: We don't set an upper limit for something we are going to malloc. Earlier it was atleast limited to INT_MAX. Now its LONG_MAX. I'd rather keep typecasting the long to int.
        Otherwise +1. Please change that and I'm happy to commit

        Show
        raviprak Ravi Prakash added a comment - Nitpick: We don't set an upper limit for something we are going to malloc. Earlier it was atleast limited to INT_MAX. Now its LONG_MAX. I'd rather keep typecasting the long to int. Otherwise +1. Please change that and I'm happy to commit
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 5m 13s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 32s There were no new javac warning messages.
        +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 yarn tests 6m 0s Tests passed in hadoop-yarn-server-nodemanager.
            21m 17s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731625/YARN-1519.002.patch
        Optional Tests javac unit
        git revision trunk / 6471d18
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7837/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7837/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7837/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 32s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 yarn tests 6m 0s Tests passed in hadoop-yarn-server-nodemanager.     21m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731625/YARN-1519.002.patch Optional Tests javac unit git revision trunk / 6471d18 hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7837/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7837/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7837/console This message was automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        Hi Ravi Prakash. Thank you for reviewing this issue.

        Nitpick: We don't set an upper limit for something we are going to malloc. Earlier it was atleast limited to INT_MAX. Now its LONG_MAX. I'd rather keep typecasting the long to int.

        Radim Kolar, do you want to make that change or should I?

        Show
        eepayne Eric Payne added a comment - Hi Ravi Prakash . Thank you for reviewing this issue. Nitpick: We don't set an upper limit for something we are going to malloc. Earlier it was atleast limited to INT_MAX. Now its LONG_MAX. I'd rather keep typecasting the long to int. Radim Kolar , do you want to make that change or should I?
        Hide
        eepayne Eric Payne added a comment -

        Radim Kolar, since I didn't hear back, I will go ahead and post the patch with the changes suggested by Ravi Prakash. Thanks again for doing all of the work and testing on this patch.

        Ravi Prakash, will you please have a look? Thanks.

        Show
        eepayne Eric Payne added a comment - Radim Kolar , since I didn't hear back, I will go ahead and post the patch with the changes suggested by Ravi Prakash . Thanks again for doing all of the work and testing on this patch. Ravi Prakash , will you please have a look? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 5m 14s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 29s There were no new javac warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 yarn tests 6m 9s Tests passed in hadoop-yarn-server-nodemanager.
            21m 23s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12732710/YARN-1519.003.patch
        Optional Tests javac unit
        git revision trunk / 281d47a
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7928/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7928/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7928/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 14s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 29s There were no new javac warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 yarn tests 6m 9s Tests passed in hadoop-yarn-server-nodemanager.     21m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732710/YARN-1519.003.patch Optional Tests javac unit git revision trunk / 281d47a hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7928/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7928/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7928/console This message was automatically generated.
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks for the patch Eric and Radim. +1. lgtm. Will check in shortly

        Show
        raviprak Ravi Prakash added a comment - Thanks for the patch Eric and Radim. +1. lgtm. Will check in shortly
        Hide
        raviprak Ravi Prakash added a comment -

        I've committed the patch to branch-2 and trunk. Thanks Eric and Radim

        Show
        raviprak Ravi Prakash added a comment - I've committed the patch to branch-2 and trunk. Thanks Eric and Radim
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7835 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7835/)
        YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7835 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7835/ ) YARN-1519 . Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #928 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/928/)
        YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #928 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/928/ ) YARN-1519 . Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #197 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/197/)
        YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #197 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/197/ ) YARN-1519 . Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2126 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2126/)
        YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2126 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2126/ ) YARN-1519 . Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/186/)
        YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/186/ ) YARN-1519 . Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/196/)
        YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/196/ ) YARN-1519 . Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2144 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2144/)
        YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2144 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2144/ ) YARN-1519 . Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak) (raviprak: rev 53fe4eff09fdaeed75a8cad3a26156bf963a8d37) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt

          People

          • Assignee:
            hsn Radim Kolar
            Reporter:
            hsn Radim Kolar
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development