Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6721

container-executor should have stack checking

    Details

      Description

      As per https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt and given that container-executor is setuid, it should be compiled with stack checking if the compiler supports such features. (-fstack-check on gcc, -fsanitize=safe-stack on clang, -xcheck=stkovf on "Oracle Solaris Studio", others as we find them, ...)

      1. YARN-6721.00.patch
        2 kB
        Allen Wittenauer
      2. YARN-6721.01.patch
        4 kB
        Allen Wittenauer
      3. YARN-6721.02.patch
        4 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12289 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12289/)
          YARN-6721. container-executor should have stack checking (aw: rev 0adc3a0533e90c8a42c5924be4847753e7f8d281)

          • (edit) hadoop-common-project/hadoop-common/HadoopCommon.cmake
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12289 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12289/ ) YARN-6721 . container-executor should have stack checking (aw: rev 0adc3a0533e90c8a42c5924be4847753e7f8d281) (edit) hadoop-common-project/hadoop-common/HadoopCommon.cmake (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
          Hide
          aw Allen Wittenauer added a comment -

          Thanks!

          Committed to trunk!

          Show
          aw Allen Wittenauer added a comment - Thanks! Committed to trunk!
          Hide
          chris.douglas Chris Douglas added a comment -

          Cool, ship it. +1

          Show
          chris.douglas Chris Douglas added a comment - Cool, ship it. +1
          Hide
          aw Allen Wittenauer added a comment -

          OK, yeah, this one is working on Linux and OS X with both gcc and clang, from what I've seen.

          Show
          aw Allen Wittenauer added a comment - OK, yeah, this one is working on Linux and OS X with both gcc and clang, from what I've seen.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 3m 37s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 13m 49s trunk passed
          +1 compile 14m 25s trunk passed
          +1 mvnsite 2m 3s trunk passed
                Patch Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 3s the patch passed
          +1 compile 10m 44s the patch passed
          +1 cc 10m 44s the patch passed
          +1 javac 10m 44s the patch passed
          +1 mvnsite 1m 59s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
                Other Tests
          +1 unit 8m 35s hadoop-common in the patch passed.
          +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          71m 47s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6721
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884786/YARN-6721.02.patch
          Optional Tests asflicense compile cc mvnsite javac unit
          uname Linux efffb9e9ac5c 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d4417da
          Default Java 1.8.0_144
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17241/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17241/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 3m 37s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 13m 49s trunk passed +1 compile 14m 25s trunk passed +1 mvnsite 2m 3s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 3s the patch passed +1 compile 10m 44s the patch passed +1 cc 10m 44s the patch passed +1 javac 10m 44s the patch passed +1 mvnsite 1m 59s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues.       Other Tests +1 unit 8m 35s hadoop-common in the patch passed. +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 71m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6721 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884786/YARN-6721.02.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux efffb9e9ac5c 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4417da Default Java 1.8.0_144 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17241/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/17241/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 14m 23s trunk passed
          +1 compile 15m 27s trunk passed
          +1 mvnsite 2m 6s trunk passed
                Patch Compile Tests
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 4s the patch passed
          +1 compile 11m 47s the patch passed
          +1 cc 11m 47s the patch passed
          +1 javac 11m 47s the patch passed
          +1 mvnsite 2m 1s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
                Other Tests
          -1 unit 8m 11s hadoop-common in the patch failed.
          +1 unit 14m 41s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          71m 52s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6721
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884784/YARN-6721.01.patch
          Optional Tests asflicense compile cc mvnsite javac unit
          uname Linux c758b7b989c8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d4417da
          Default Java 1.8.0_144
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17240/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17240/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17240/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 14m 23s trunk passed +1 compile 15m 27s trunk passed +1 mvnsite 2m 6s trunk passed       Patch Compile Tests 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 11m 47s the patch passed +1 cc 11m 47s the patch passed +1 javac 11m 47s the patch passed +1 mvnsite 2m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues.       Other Tests -1 unit 8m 11s hadoop-common in the patch failed. +1 unit 14m 41s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 71m 52s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6721 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884784/YARN-6721.01.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux c758b7b989c8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4417da Default Java 1.8.0_144 unit https://builds.apache.org/job/PreCommit-YARN-Build/17240/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17240/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/17240/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -02:

          • fix GNU C++ flags
          Show
          aw Allen Wittenauer added a comment - -02: fix GNU C++ flags
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • fix up some clang issues
          • add -pthread in a more appropriate manner
          • gtest needs the flags too
          Show
          aw Allen Wittenauer added a comment - -01: fix up some clang issues add -pthread in a more appropriate manner gtest needs the flags too
          Hide
          aw Allen Wittenauer added a comment -

          Thanks, but after a bit more testing, clang 4.0 on Linux is blowing up. Looks like a trivial fix though.

          Show
          aw Allen Wittenauer added a comment - Thanks, but after a bit more testing, clang 4.0 on Linux is blowing up. Looks like a trivial fix though.
          Hide
          chris.douglas Chris Douglas added a comment -

          Bravo, figuring out what's is going on with clang. I looked for supporting documentation on OSX, and found mostly confusion.

          +1

          Show
          chris.douglas Chris Douglas added a comment - Bravo, figuring out what's is going on with clang. I looked for supporting documentation on OSX, and found mostly confusion. +1
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 19m 5s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          +1 mvninstall 13m 56s trunk passed
          +1 compile 0m 42s trunk passed
          +1 mvnsite 0m 28s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 40s the patch passed
          +1 cc 0m 40s the patch passed
          +1 javac 0m 40s the patch passed
          +1 mvnsite 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
                Other Tests
          +1 unit 14m 21s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 13s The patch does not generate ASF License warnings.
          50m 39s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6721
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884515/YARN-6721.00.patch
          Optional Tests asflicense compile cc mvnsite javac unit
          uname Linux 9434ec6b846c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 71bbb86
          Default Java 1.8.0_144
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17225/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17225/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 19m 5s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 13m 56s trunk passed +1 compile 0m 42s trunk passed +1 mvnsite 0m 28s trunk passed       Patch Compile Tests +1 mvninstall 0m 25s the patch passed +1 compile 0m 40s the patch passed +1 cc 0m 40s the patch passed +1 javac 0m 40s the patch passed +1 mvnsite 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues.       Other Tests +1 unit 14m 21s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 13s The patch does not generate ASF License warnings. 50m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6721 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884515/YARN-6721.00.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux 9434ec6b846c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 71bbb86 Default Java 1.8.0_144 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17225/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/17225/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -00:

          • support for gcc, clang, and Sun
          • expects HADOOP-14670 to be applied first
          Show
          aw Allen Wittenauer added a comment - -00: support for gcc, clang, and Sun expects HADOOP-14670 to be applied first
          Hide
          shv Konstantin Shvachko added a comment -

          Moving target version to 2.7.5 due to 2.7.4 release.

          Show
          shv Konstantin Shvachko added a comment - Moving target version to 2.7.5 due to 2.7.4 release.
          Hide
          sunilg Sunil G added a comment -

          I would like to take it up and give it a go. Please assign back if any one is interested. I will share a patch soon.

          Show
          sunilg Sunil G added a comment - I would like to take it up and give it a go. Please assign back if any one is interested. I will share a patch soon.
          Hide
          aw Allen Wittenauer added a comment -

          -fsanitize=safe-stack on clang

          This doesn't work on OS X because Apple doesn't ship the SafeStack runtime as part of Xcode. I've filed a Radar so maybe at some point in the next 5 years it'll be able to be used. :/

          Show
          aw Allen Wittenauer added a comment - -fsanitize=safe-stack on clang This doesn't work on OS X because Apple doesn't ship the SafeStack runtime as part of Xcode. I've filed a Radar so maybe at some point in the next 5 years it'll be able to be used. :/

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development