Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6757

Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.1, 3.0.0-alpha4
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: nodemanager
    • Labels:
      None

      Description

      We should add the ability to specify a custom cgroup path. This is how the documentation of linux-container-executor.cgroups.mount-path would look like:

          Requested cgroup mount path. Yarn has built in functionality to discover
          the system cgroup mount paths, so use this setting only, if the discovery does not work.
      
          This path must exist before the NodeManager is launched.
          The location can vary depending on the Linux distribution in use.
          Common locations include /sys/fs/cgroup and /cgroup.
      
          If cgroups are not mounted, set yarn.nodemanager.linux-container-executor.cgroups.mount
          to true. In this case it specifies, where the LCE should attempt to mount cgroups if not found.
      
          If cgroups is accessible through lxcfs or some other file system,
          then set this path and yarn.nodemanager.linux-container-executor.cgroups.mount to false.
          Yarn tries to use this path first, before any cgroup mount point discovery.
          If it cannot find this directory, it falls back to searching for cgroup mount points in the system.
          Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler
      
      1. YARN-6757.000.patch
        16 kB
        Miklos Szegedi
      2. YARN-6757.001.patch
        16 kB
        Yufei Gu
      3. YARN-6757.002.patch
        28 kB
        Miklos Szegedi
      4. YARN-6757.003.patch
        28 kB
        Miklos Szegedi
      5. YARN-6757.004.patch
        57 kB
        Miklos Szegedi
      6. YARN-6757.005.patch
        39 kB
        Miklos Szegedi

        Issue Links

          Activity

          Hide
          templedf Daniel Templeton added a comment -

          Also committed to branch-2.

          Show
          templedf Daniel Templeton added a comment - Also committed to branch-2.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Yes please.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Yes please.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the patch, Miklos Szegedi. Committed to trunk. Did you also want to push to branch-2?

          Show
          templedf Daniel Templeton added a comment - Thanks for the patch, Miklos Szegedi . Committed to trunk. Did you also want to push to branch-2?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 41s Maven dependency ordering for branch
          +1 mvninstall 14m 12s trunk passed
          +1 compile 9m 22s trunk passed
          +1 checkstyle 0m 58s trunk passed
          +1 mvnsite 1m 38s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 1m 25s trunk passed
                Patch Compile Tests
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 7s the patch passed
          +1 compile 5m 39s the patch passed
          +1 javac 5m 39s the patch passed
          +1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23)
          +1 mvnsite 1m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 2m 26s the patch passed
          +1 javadoc 1m 23s the patch passed
                Other Tests
          +1 unit 2m 33s hadoop-yarn-common in the patch passed.
          +1 unit 13m 28s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 0m 16s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          68m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6757
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880306/YARN-6757.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 6e2264125504 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f4c6b00
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16701/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16701/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16701/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 41s Maven dependency ordering for branch +1 mvninstall 14m 12s trunk passed +1 compile 9m 22s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 1m 38s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 1m 25s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 7s the patch passed +1 compile 5m 39s the patch passed +1 javac 5m 39s the patch passed +1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) +1 mvnsite 1m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 2m 26s the patch passed +1 javadoc 1m 23s the patch passed       Other Tests +1 unit 2m 33s hadoop-yarn-common in the patch passed. +1 unit 13m 28s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 16s hadoop-yarn-site in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 68m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6757 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880306/YARN-6757.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 6e2264125504 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f4c6b00 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16701/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16701/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16701/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          LGTM. +1

          Show
          templedf Daniel Templeton added a comment - LGTM. +1
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          I attached the patch.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - I attached the patch.
          Hide
          templedf Daniel Templeton added a comment -

          Fine, leave Yarn in there and file a follow up JIRA to make it YARN in all the docs.

          Show
          templedf Daniel Templeton added a comment - Fine, leave Yarn in there and file a follow up JIRA to make it YARN in all the docs.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Hmm, I do not understand. If YARN is the official naming should not we fix them everywhere now and enforce from now on? If the rule is just for this change, I actually think Yarn is more readable.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Hmm, I do not understand. If YARN is the official naming should not we fix them everywhere now and enforce from now on? If the rule is just for this change, I actually think Yarn is more readable.
          Hide
          templedf Daniel Templeton added a comment -

          I think the YARN replacement may have gone a bit too far. I don't think you should be making changes in the release notes for shipped releases. To keep this from being a huge patch, let's limit the changes to the md doc you're editing and the text you're adding to yarn-defaults.xml.

          Show
          templedf Daniel Templeton added a comment - I think the YARN replacement may have gone a bit too far. I don't think you should be making changes in the release notes for shipped releases. To keep this from being a huge patch, let's limit the changes to the md doc you're editing and the text you're adding to yarn-defaults.xml.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 13m 26s trunk passed
          +1 compile 13m 49s trunk passed
          +1 checkstyle 1m 56s trunk passed
          +1 mvnsite 5m 2s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
          -1 findbugs 1m 43s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          -1 findbugs 0m 44s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 3m 21s trunk passed
                Patch Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 3m 30s the patch passed
          +1 compile 11m 33s the patch passed
          +1 javac 11m 33s the patch passed
          +1 checkstyle 1m 59s root: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23)
          +1 mvnsite 5m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
          +1 findbugs 8m 7s the patch passed
          +1 javadoc 4m 7s the patch passed
                Other Tests
          +1 unit 8m 25s hadoop-common in the patch passed.
          -1 unit 74m 25s hadoop-hdfs in the patch failed.
          +1 unit 2m 39s hadoop-yarn-common in the patch passed.
          +1 unit 13m 22s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 0m 15s hadoop-yarn-site in the patch passed.
          +1 unit 3m 8s hadoop-yarn-ui in the patch passed.
          +1 unit 0m 51s hadoop-archive-logs in the patch passed.
          +1 unit 8m 10s hadoop-sls in the patch passed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          213m 27s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6757
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879709/YARN-6757.004.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle
          uname Linux 43d67b7d6e00 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b38a1ee
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16645/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16645/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16645/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16645/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16645/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 13m 26s trunk passed +1 compile 13m 49s trunk passed +1 checkstyle 1m 56s trunk passed +1 mvnsite 5m 2s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui -1 findbugs 1m 43s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. -1 findbugs 0m 44s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 3m 21s trunk passed       Patch Compile Tests 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 3m 30s the patch passed +1 compile 11m 33s the patch passed +1 javac 11m 33s the patch passed +1 checkstyle 1m 59s root: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) +1 mvnsite 5m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui +1 findbugs 8m 7s the patch passed +1 javadoc 4m 7s the patch passed       Other Tests +1 unit 8m 25s hadoop-common in the patch passed. -1 unit 74m 25s hadoop-hdfs in the patch failed. +1 unit 2m 39s hadoop-yarn-common in the patch passed. +1 unit 13m 22s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 15s hadoop-yarn-site in the patch passed. +1 unit 3m 8s hadoop-yarn-ui in the patch passed. +1 unit 0m 51s hadoop-archive-logs in the patch passed. +1 unit 8m 10s hadoop-sls in the patch passed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 213m 27s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6757 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879709/YARN-6757.004.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle uname Linux 43d67b7d6e00 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b38a1ee Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16645/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16645/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16645/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16645/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/16645/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          I changed only documentation in the recent patch, so the unit test failures are probably due to something else. Can we rerun?

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - I changed only documentation in the recent patch, so the unit test failures are probably due to something else. Can we rerun?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 19m 6s trunk passed
          +1 compile 19m 44s trunk passed
          +1 checkstyle 1m 58s trunk passed
          +1 mvnsite 5m 45s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
          -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 3m 57s trunk passed
                Patch Compile Tests
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 4m 14s the patch passed
          +1 compile 14m 30s the patch passed
          +1 javac 14m 30s the patch passed
          +1 checkstyle 2m 21s root: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23)
          +1 mvnsite 7m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
          +1 findbugs 8m 51s the patch passed
          +1 javadoc 3m 45s the patch passed
                Other Tests
          +1 unit 9m 2s hadoop-common in the patch passed.
          -1 unit 106m 58s hadoop-hdfs in the patch failed.
          +1 unit 2m 49s hadoop-yarn-common in the patch passed.
          +1 unit 14m 7s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 0m 16s hadoop-yarn-site in the patch passed.
          +1 unit 5m 27s hadoop-yarn-ui in the patch passed.
          +1 unit 1m 30s hadoop-archive-logs in the patch passed.
          +1 unit 8m 21s hadoop-sls in the patch passed.
          +1 asflicense 0m 46s The patch does not generate ASF License warnings.
          271m 18s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
            hadoop.hdfs.server.namenode.ha.TestHAMetrics
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6757
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879709/YARN-6757.004.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle
          uname Linux 75596d1f589c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3e23415
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16627/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16627/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16627/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16627/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16627/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 19m 6s trunk passed +1 compile 19m 44s trunk passed +1 checkstyle 1m 58s trunk passed +1 mvnsite 5m 45s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 3m 57s trunk passed       Patch Compile Tests 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 4m 14s the patch passed +1 compile 14m 30s the patch passed +1 javac 14m 30s the patch passed +1 checkstyle 2m 21s root: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) +1 mvnsite 7m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui +1 findbugs 8m 51s the patch passed +1 javadoc 3m 45s the patch passed       Other Tests +1 unit 9m 2s hadoop-common in the patch passed. -1 unit 106m 58s hadoop-hdfs in the patch failed. +1 unit 2m 49s hadoop-yarn-common in the patch passed. +1 unit 14m 7s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 16s hadoop-yarn-site in the patch passed. +1 unit 5m 27s hadoop-yarn-ui in the patch passed. +1 unit 1m 30s hadoop-archive-logs in the patch passed. +1 unit 8m 21s hadoop-sls in the patch passed. +1 asflicense 0m 46s The patch does not generate ASF License warnings. 271m 18s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.namenode.ha.TestHAMetrics   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6757 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879709/YARN-6757.004.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle uname Linux 75596d1f589c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3e23415 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16627/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16627/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16627/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16627/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/16627/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Thank you. I attached the update.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you. I attached the update.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the update, Miklos Szegedi. A few more comments, mostly language/grammar:

          • The ResourceHandlerModule methods (getTrafficControlBandwidthHandler, et al) where you fixed the indentation should now have the third line indented two more spaces.
          • In yarn-defaults.xml:
            • "so use this setting only, if the discovery does not work" shouldn't have a comma.
            • Yarn should be YARN.
            • I really struggle with this whole text. What about something like: "This property sets the path from which YARN will read the CGroups configuration. YARN has built-in functionality to discover the system CGroup mount paths, so use this property only if YARN's automatic mount path discovery does not work. The path specified by this property must exist before the NodeManager is launched. If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true, YARN will first try to mount the CGroups at the specified path before reading them. Please refer to NodeManagerCgroups.html in the documentation for further details. This property only applies when the LCE resources handler is set to CgroupsLCEResourcesHandler."
          • In NodeManagerCgroups.md:
            • In the definition of yarn.nodemanager.linux-container-executor.cgroups.mount-path, I think you're better off just saying, "Optional. See CGroups mount options below for details."
            • Yarn should be YARN.
            • "leave other settings default" => "leave other settings set to the defaults"
            • "Mount CGroups by Yarn" => "CGroups mounted by YARN"
            • "CGroups mounted or linked already but not in `/proc/mounts`" => "CGroups already mounted or linked but not in `/proc/mounts`"
          Show
          templedf Daniel Templeton added a comment - Thanks for the update, Miklos Szegedi . A few more comments, mostly language/grammar: The ResourceHandlerModule methods ( getTrafficControlBandwidthHandler , et al) where you fixed the indentation should now have the third line indented two more spaces. In yarn-defaults.xml : "so use this setting only, if the discovery does not work" shouldn't have a comma. Yarn should be YARN. I really struggle with this whole text. What about something like: "This property sets the path from which YARN will read the CGroups configuration. YARN has built-in functionality to discover the system CGroup mount paths, so use this property only if YARN's automatic mount path discovery does not work. The path specified by this property must exist before the NodeManager is launched. If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true, YARN will first try to mount the CGroups at the specified path before reading them. Please refer to NodeManagerCgroups.html in the documentation for further details. This property only applies when the LCE resources handler is set to CgroupsLCEResourcesHandler." In NodeManagerCgroups.md : In the definition of yarn.nodemanager.linux-container-executor.cgroups.mount-path, I think you're better off just saying, "Optional. See CGroups mount options below for details." Yarn should be YARN. "leave other settings default" => "leave other settings set to the defaults" "Mount CGroups by Yarn" => "CGroups mounted by YARN" "CGroups mounted or linked already but not in `/proc/mounts`" => "CGroups already mounted or linked but not in `/proc/mounts`"
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 13m 37s trunk passed
          +1 compile 8m 44s trunk passed
          +1 checkstyle 0m 54s trunk passed
          +1 mvnsite 1m 36s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 1m 21s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 9s the patch passed
          +1 compile 5m 20s the patch passed
          +1 javac 5m 20s the patch passed
          +1 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23)
          +1 mvnsite 1m 33s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 2m 30s the patch passed
          +1 javadoc 1m 22s the patch passed
                Other Tests
          +1 unit 2m 30s hadoop-yarn-common in the patch passed.
          +1 unit 13m 25s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 0m 15s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          66m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6757
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879435/YARN-6757.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux f29b42ace7fe 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 713349a
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16601/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16601/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16601/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 13m 37s trunk passed +1 compile 8m 44s trunk passed +1 checkstyle 0m 54s trunk passed +1 mvnsite 1m 36s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 1m 21s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 5m 20s the patch passed +1 javac 5m 20s the patch passed +1 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) +1 mvnsite 1m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 2m 30s the patch passed +1 javadoc 1m 22s the patch passed       Other Tests +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 unit 13m 25s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 15s hadoop-yarn-site in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 66m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6757 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879435/YARN-6757.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux f29b42ace7fe 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 713349a Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16601/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16601/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16601/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 14m 41s trunk passed
          +1 compile 8m 55s trunk passed
          +1 checkstyle 0m 54s trunk passed
          +1 mvnsite 1m 38s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          -1 findbugs 0m 54s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 1m 28s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 5m 42s the patch passed
          +1 javac 5m 42s the patch passed
          +1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23)
          +1 mvnsite 1m 39s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 2m 32s the patch passed
          +1 javadoc 1m 22s the patch passed
                Other Tests
          +1 unit 2m 27s hadoop-yarn-common in the patch passed.
          +1 unit 13m 11s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 0m 15s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          68m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6757
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879415/YARN-6757.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 63f6f26503c0 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 746189a
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16597/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16597/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16597/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16597/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 14m 41s trunk passed +1 compile 8m 55s trunk passed +1 checkstyle 0m 54s trunk passed +1 mvnsite 1m 38s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 54s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 1m 28s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 5m 42s the patch passed +1 javac 5m 42s the patch passed +1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) +1 mvnsite 1m 39s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 2m 32s the patch passed +1 javadoc 1m 22s the patch passed       Other Tests +1 unit 2m 27s hadoop-yarn-common in the patch passed. +1 unit 13m 11s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 15s hadoop-yarn-site in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 68m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6757 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879415/YARN-6757.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 63f6f26503c0 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 746189a Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16597/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16597/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16597/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16597/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Thanks. I updated the documentation including the html. I think that is a better way to describe the details than the xml which is not visible for everyone. I left a pointer in the xml. I did not cache getValidCGroups(), since it is just used one. Caching it would waste memory, although not much.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thanks. I updated the documentation including the html. I think that is a better way to describe the details than the xml which is not visible for everyone. I left a pointer in the xml. I did not cache getValidCGroups() , since it is just used one. Caching it would waste memory, although not much.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks. A few more comments:

          • CGroupsHandler.getValidCGroups() might profit from caching the result; it never changes.
          • In CGroupsHandlerImpl.testManualCgroupSetting() and TestCgroupsLCEResourcesHandler.testManualCgroupSetting(), it would be nice to add a message to the assertEquals().

          Otherwise, looks good. I still don't really understand what it's doing, though, so lets work on the property description, and hopefully once that's clear it'll all make sense.

          Show
          templedf Daniel Templeton added a comment - Thanks. A few more comments: CGroupsHandler.getValidCGroups() might profit from caching the result; it never changes. In CGroupsHandlerImpl.testManualCgroupSetting() and TestCgroupsLCEResourcesHandler.testManualCgroupSetting() , it would be nice to add a message to the assertEquals() . Otherwise, looks good. I still don't really understand what it's doing, though, so lets work on the property description, and hopefully once that's clear it'll all make sense.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          1 @author 0m 0s The patch does not contain any @author tags.
          1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          1 mvninstall 13m 7s trunk passed
          1 compile 8m 26s trunk passed
          1 checkstyle 0m 50s trunk passed
          1 mvnsite 1m 9s trunk passed
          -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          1 javadoc 1m 0s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          1 mvninstall 0m 52s the patch passed
          1 compile 5m 38s the patch passed
          1 javac 5m 38s the patch passed
          1 checkstyle 1m 4s the patch passed
          1 mvnsite 1m 17s the patch passed
          1 whitespace 0m 0s The patch has no whitespace issues.
          1 xml 0m 2s The patch has no ill-formed XML file.
          1 findbugs 2m 20s the patch passed
          1 javadoc 0m 56s the patch passed
          1 unit 2m 32s hadoop-yarn-common in the patch passed.
          1 unit 13m 17s hadoop-yarn-server-nodemanager in the patch passed.
          1 asflicense 0m 30s The patch does not generate ASF License warnings.
          62m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6757
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875999/YARN-6757.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux a0a30a72ee99 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7576a68
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16318/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16318/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16318/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. 1 @author 0m 0s The patch does not contain any @author tags. 1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch 1 mvninstall 13m 7s trunk passed 1 compile 8m 26s trunk passed 1 checkstyle 0m 50s trunk passed 1 mvnsite 1m 9s trunk passed -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. 1 javadoc 1m 0s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch 1 mvninstall 0m 52s the patch passed 1 compile 5m 38s the patch passed 1 javac 5m 38s the patch passed 1 checkstyle 1m 4s the patch passed 1 mvnsite 1m 17s the patch passed 1 whitespace 0m 0s The patch has no whitespace issues. 1 xml 0m 2s The patch has no ill-formed XML file. 1 findbugs 2m 20s the patch passed 1 javadoc 0m 56s the patch passed 1 unit 2m 32s hadoop-yarn-common in the patch passed. 1 unit 13m 17s hadoop-yarn-server-nodemanager in the patch passed. 1 asflicense 0m 30s The patch does not generate ASF License warnings. 62m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6757 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875999/YARN-6757.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux a0a30a72ee99 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7576a68 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16318/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16318/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16318/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Uploaded a new patch.
          1. fixed.
          2. CGroupsHandler.getValidCGroups() should be OK since EnumSet keep the enum instead of its name string. It's easier for callers to use name string HashSet.
          3. 4. 5. 6. fixed.
          7. Yes, it's weird, but we cannot do anything about it.
          8. Fixed partially. Need the offline discussion, then I can post a new patch.

          Show
          yufeigu Yufei Gu added a comment - Uploaded a new patch. 1. fixed. 2. CGroupsHandler.getValidCGroups() should be OK since EnumSet keep the enum instead of its name string. It's easier for callers to use name string HashSet. 3. 4. 5. 6. fixed. 7. Yes, it's weird, but we cannot do anything about it. 8. Fixed partially. Need the offline discussion, then I can post a new patch.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Daniel Templeton's review. I will take this as the offline discussion with Miklos Szegedi. Will post the new patch soon.

          Show
          yufeigu Yufei Gu added a comment - Thanks Daniel Templeton 's review. I will take this as the offline discussion with Miklos Szegedi . Will post the new patch soon.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the patch, Miklos Szegedi. A few comments:

          1. The javadoc description for CGroupsHandler.getValidCGroups() should end with a period.
          2. CGroupsHandler.getValidCGroups() seems altogether unnecessary. It's really just doing new HashSet<>(Arrays.asList(CGroupsController.values())) long hand. Maybe even better would be to use an EnumSet instead of a HashSet.
          3. In {checkConfiguredCGroupPath()}}, cGroupMountPathSpecified is a little long-winded. cGroupMountPath is plenty.
          4. checkConfiguredCGroupPath() isn't a great name. Maybe loadConfiguredCGroupPath() or parseConfiguredCGroupPath?
          5. Why does CgroupsLCEResourcesHandler have an identical copy of checkConfiguredCGroupPath()? Seems like there should be some code sharing...
          6. Theres now a dead import in CgroupsLCEResourcesHandler.
          7. What's with the mount path containing directories that contain files whose names are comma-separated lists or cgroups? Is that a normal cgroups thing? Sounds weird to me...
          8. I love that the text in yarn-defaults.xml is detailed and prescriptive, but it needs some work. There are grammar issues, but first let's tackle the clarity issues. I don't know what it's trying to say. I get that the property sets a path that we'll use to resolve cgroups under certain circumstances, but it's not clear what those circumstances are and what will happen when this property is set. It also says nothing about the file names that are comma-separated lists of cgroups. If it helps, I'm happy to have an offline conversation and help craft the docs.
          Show
          templedf Daniel Templeton added a comment - Thanks for the patch, Miklos Szegedi . A few comments: The javadoc description for CGroupsHandler.getValidCGroups() should end with a period. CGroupsHandler.getValidCGroups() seems altogether unnecessary. It's really just doing new HashSet<>(Arrays.asList(CGroupsController.values())) long hand. Maybe even better would be to use an EnumSet instead of a HashSet . In {checkConfiguredCGroupPath()}}, cGroupMountPathSpecified is a little long-winded. cGroupMountPath is plenty. checkConfiguredCGroupPath() isn't a great name. Maybe loadConfiguredCGroupPath() or parseConfiguredCGroupPath ? Why does CgroupsLCEResourcesHandler have an identical copy of checkConfiguredCGroupPath() ? Seems like there should be some code sharing... Theres now a dead import in CgroupsLCEResourcesHandler . What's with the mount path containing directories that contain files whose names are comma-separated lists or cgroups? Is that a normal cgroups thing? Sounds weird to me... I love that the text in yarn-defaults.xml is detailed and prescriptive, but it needs some work. There are grammar issues, but first let's tackle the clarity issues. I don't know what it's trying to say. I get that the property sets a path that we'll use to resolve cgroups under certain circumstances, but it's not clear what those circumstances are and what will happen when this property is set. It also says nothing about the file names that are comma-separated lists of cgroups. If it helps, I'm happy to have an offline conversation and help craft the docs.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          See also the discussion in YARN-6515, why we need this.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - See also the discussion in YARN-6515 , why we need this.

            People

            • Assignee:
              miklos.szegedi@cloudera.com Miklos Szegedi
              Reporter:
              miklos.szegedi@cloudera.com Miklos Szegedi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development