Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3427

Remove deprecated methods from ResourceCalculatorProcessTree

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The deprecated ProcessTree methods getCumulativeVmem
       and getCumulativeRssmem have been removed.
      Show
      The deprecated ProcessTree methods getCumulativeVmem  and getCumulativeRssmem have been removed.

      Description

      In 2.7, we made ResourceCalculatorProcessTree Public and exposed some existing ill-formed methods as deprecated ones for use by Tez.

      We should remove it in 3.0.0, considering that the methods have been deprecated for the all 2.x.y releases that it is marked Public in.

      1. YARN-3427.000.patch
        16 kB
        Miklos Szegedi
      2. YARN-3427.001.patch
        16 kB
        Miklos Szegedi

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Added a release note for this incompatible change, please update if it's incorrect.

          Show
          andrew.wang Andrew Wang added a comment - Added a release note for this incompatible change, please update if it's incorrect.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11607 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11607/)
          YARN-3427. Remove deprecated methods from ResourceCalculatorProcessTree (templedf: rev 15acdbdba26c8c869c7dcd2fa398af5427735798)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsBasedProcessTree.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWindowsBasedProcessTree.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11607 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11607/ ) YARN-3427 . Remove deprecated methods from ResourceCalculatorProcessTree (templedf: rev 15acdbdba26c8c869c7dcd2fa398af5427735798) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsBasedProcessTree.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWindowsBasedProcessTree.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the patch, Miklos Szegedi. Committed to trunk.

          Show
          templedf Daniel Templeton added a comment - Thanks for the patch, Miklos Szegedi . Committed to trunk.
          Hide
          templedf Daniel Templeton added a comment -

          Heads up, Siddharth Seth, Rajesh Balamohan, Hitesh Shah, I'm planning to commit this JIRA this week.

          Show
          templedf Daniel Templeton added a comment - Heads up, Siddharth Seth , Rajesh Balamohan , Hitesh Shah , I'm planning to commit this JIRA this week.
          Hide
          templedf Daniel Templeton added a comment -

          The methods were made public in 2.7 specifically for Tez, but they were exposed as deprecated with the very explicit plan of removing them in 3.0.

          Show
          templedf Daniel Templeton added a comment - The methods were made public in 2.7 specifically for Tez, but they were exposed as deprecated with the very explicit plan of removing them in 3.0.
          Hide
          sseth Siddharth Seth added a comment -

          From a Tez perspective, would prefer if the methods were left in place. If this was something that was fixed in 2.6, that would have been easier to work with. Since the new methods were in 2.7 - Tez will need to introduce a shim for this.

          Show
          sseth Siddharth Seth added a comment - From a Tez perspective, would prefer if the methods were left in place. If this was something that was fixed in 2.6, that would have been easier to work with. Since the new methods were in 2.7 - Tez will need to introduce a shim for this.
          Hide
          hitesh Hitesh Shah added a comment -

          Thanks for the heads up Daniel Templeton. \cc Siddharth Seth Rajesh Balamohan

          Show
          hitesh Hitesh Shah added a comment - Thanks for the heads up Daniel Templeton . \cc Siddharth Seth Rajesh Balamohan
          Hide
          templedf Daniel Templeton added a comment -

          LGTM +1 I'll commit it next week. CC: Hitesh Shah

          Show
          templedf Daniel Templeton added a comment - LGTM +1 I'll commit it next week. CC: Hitesh Shah
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 13m 35s trunk passed
          +1 compile 0m 27s trunk passed
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 30s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 57s trunk passed
          +1 javadoc 0m 29s trunk passed
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 27s the patch passed
          +1 javac 0m 27s the patch passed
          -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new + 126 unchanged - 21 fixed = 127 total (was 147)
          +1 mvnsite 0m 29s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 9s the patch passed
          +1 javadoc 0m 27s the patch passed
          +1 unit 2m 21s hadoop-yarn-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          24m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-3427
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860462/YARN-3427.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 76dac8689952 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1f66524
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15383/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15383/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15383/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 35s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 29s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new + 126 unchanged - 21 fixed = 127 total (was 147) +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 9s the patch passed +1 javadoc 0m 27s the patch passed +1 unit 2m 21s hadoop-yarn-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 24m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3427 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860462/YARN-3427.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 76dac8689952 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1f66524 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15383/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15383/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/15383/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Good point. I updated the patch.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Good point. I updated the patch.
          Hide
          templedf Daniel Templeton added a comment -

          Is it now safe to also remove the suppression on the testMemForOlderProcesses() method?

          Show
          templedf Daniel Templeton added a comment - Is it now safe to also remove the suppression on the testMemForOlderProcesses() method?
          Hide
          yufeigu Yufei Gu added a comment -

          +1 (non-binding)

          Show
          yufeigu Yufei Gu added a comment - +1 (non-binding)
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 14m 20s trunk passed
          +1 compile 0m 33s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 6s trunk passed
          +1 javadoc 0m 29s trunk passed
          +1 mvninstall 0m 27s the patch passed
          +1 compile 0m 25s the patch passed
          +1 javac 0m 25s the patch passed
          -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new + 126 unchanged - 21 fixed = 127 total (was 147)
          +1 mvnsite 0m 29s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 5s the patch passed
          +1 javadoc 0m 27s the patch passed
          +1 unit 2m 20s hadoop-yarn-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          25m 29s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-3427
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859657/YARN-3427.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 13dfd88aa020 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 49efd5d
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15342/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15342/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15342/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 14m 20s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 6s trunk passed +1 javadoc 0m 29s trunk passed +1 mvninstall 0m 27s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new + 126 unchanged - 21 fixed = 127 total (was 147) +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 27s the patch passed +1 unit 2m 20s hadoop-yarn-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 25m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3427 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859657/YARN-3427.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 13dfd88aa020 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 49efd5d Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15342/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15342/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/15342/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Attached patch

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Attached patch

            People

            • Assignee:
              miklos.szegedi@cloudera.com Miklos Szegedi
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development