Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14840

Tool to estimate resource requirements of an application pipeline based on prior executions

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0, 3.1.0
    • Component/s: tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The first version of Resource Estimator service, a tool that captures the historical resource usage of an app and predicts its future resource requirement.

      Description

      We have been working on providing SLAs for job execution on Hadoop. At high level this involves 2 parts: deriving the resource requirements of a job and guaranteeing the estimated resources at runtime. The YARN ReservationSystem (YARN-1051/YARN-2572/YARN-5326) enable the latter and in this JIRA, we propose to add a tool to Hadoop to predict the resource requirements of a job based on past executions of the job. The system (aka Morpheus) deep dive can be found in our OSDI'16 paper here.

      1. ResourceEstimator-design-v1.pdf
        349 kB
        Subru Krishnan
      2. HADOOP-14840-v1.patch
        344 kB
        Rui Li
      3. HADOOP-14840-v2.patch
        365 kB
        Rui Li
      4. HADOOP-14840-v3.patch
        367 kB
        Rui Li
      5. HADOOP-14840-v4.patch
        373 kB
        Rui Li
      6. HADOOP-14840-branch-2-v4.patch
        373 kB
        Subru Krishnan

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13231 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13231/)
          HADOOP-15036. Update LICENSE.txt for HADOOP-14840. (asuresh) (arun suresh: rev f871b7541a5375eb117eafb9a091e4f59401231f)

          • (edit) LICENSE.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13231 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13231/ ) HADOOP-15036 . Update LICENSE.txt for HADOOP-14840 . (asuresh) (arun suresh: rev f871b7541a5375eb117eafb9a091e4f59401231f) (edit) LICENSE.txt
          Hide
          subru Subru Krishnan added a comment -

          I (re)committed it to branch-2 based on the clean Yetus report.

          Show
          subru Subru Krishnan added a comment - I (re)committed it to branch-2 based on the clean Yetus report.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 18m 32s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 31 new or modified test files.
                branch-2 Compile Tests
          0 mvndep 0m 35s Maven dependency ordering for branch
          +1 mvninstall 9m 48s branch-2 passed
          +1 compile 14m 55s branch-2 passed
          +1 checkstyle 2m 14s branch-2 passed
          +1 mvnsite 12m 3s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 0m 0s branch-2 passed
          +1 javadoc 5m 49s branch-2 passed
                Patch Compile Tests
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 9m 51s the patch passed
          +1 compile 10m 44s the patch passed
          +1 javac 10m 44s the patch passed
          +1 checkstyle 1m 46s the patch passed
          +1 mvnsite 10m 58s the patch passed
          +1 shellcheck 0m 1s There were no new shellcheck issues.
          +1 shelldocs 0m 8s There were no new shelldocs issues.
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 6m 33s the patch passed
                Other Tests
          -1 unit 29m 36s root in the patch failed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          136m 52s



          Reason Tests
          Timed out junit tests org.apache.hadoop.log.TestLogLevel



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12894195/HADOOP-14840-branch-2-v4.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle
          uname Linux d2072579225d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 4678315
          maven version: Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)
          Default Java 1.7.0_151
          shellcheck v0.4.6
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13583/artifact/out/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13583/testReport/
          modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13583/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 18m 32s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 31 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 35s Maven dependency ordering for branch +1 mvninstall 9m 48s branch-2 passed +1 compile 14m 55s branch-2 passed +1 checkstyle 2m 14s branch-2 passed +1 mvnsite 12m 3s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 0m 0s branch-2 passed +1 javadoc 5m 49s branch-2 passed       Patch Compile Tests 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 9m 51s the patch passed +1 compile 10m 44s the patch passed +1 javac 10m 44s the patch passed +1 checkstyle 1m 46s the patch passed +1 mvnsite 10m 58s the patch passed +1 shellcheck 0m 1s There were no new shellcheck issues. +1 shelldocs 0m 8s There were no new shelldocs issues. +1 whitespace 0m 1s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 0m 43s the patch passed +1 javadoc 6m 33s the patch passed       Other Tests -1 unit 29m 36s root in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 136m 52s Reason Tests Timed out junit tests org.apache.hadoop.log.TestLogLevel Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12894195/HADOOP-14840-branch-2-v4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle uname Linux d2072579225d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision branch-2 / 4678315 maven version: Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) Default Java 1.7.0_151 shellcheck v0.4.6 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13583/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13583/testReport/ modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13583/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Chris Douglas for filing HADOOP-14986. FWIW, I did run the build locally successfully after cherry-pick before pushing to branch-2.

          Show
          subru Subru Krishnan added a comment - Thanks Chris Douglas for filing HADOOP-14986 . FWIW, I did run the build locally successfully after cherry-pick before pushing to branch-2.
          Hide
          chris.douglas Chris Douglas added a comment -

          Filed HADOOP-14986 to fail the build on these kinds of errors.

          Show
          chris.douglas Chris Douglas added a comment - Filed HADOOP-14986 to fail the build on these kinds of errors.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Akira Ajisaka for bringing this to my attention. I am reopening to upload branch-2 patch to make sure there are no other issues.

          Show
          subru Subru Krishnan added a comment - Thanks Akira Ajisaka for bringing this to my attention. I am reopening to upload branch-2 patch to make sure there are no other issues.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Reverted from branch-2.

          Show
          ajisakaa Akira Ajisaka added a comment - Reverted from branch-2.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Reverting this from branch-2 because branch-2 build is broken with Java 7. The patch includes toIntExact(long), which was introduced in Java 8.

          Show
          ajisakaa Akira Ajisaka added a comment - Reverting this from branch-2 because branch-2 build is broken with Java 7. The patch includes toIntExact(long) , which was introduced in Java 8.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #13138 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13138/)
          HADOOP-14840. Tool to estimate resource requirements of an application (subru: rev 625039ef20e6011ab360131d70582a6e4bf2ec1d)

          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/solver/impl/TestSolver.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/validator/SkylineStoreValidator.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/estimator.sh
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/RecurrenceIdNotFoundException.java
          • (add) hadoop-tools/hadoop-resourceestimator/README.md
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/rmLog.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/SkylineStoreException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/exceptions/DataFieldNotFoundException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/SolverException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/site/resources/css/site.css
          • (edit) hadoop-tools/pom.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/exception/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/InvalidSolverException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/exceptions/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog8.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/EmptyResourceSkylineException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog3.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/exception/ResourceEstimatorException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/LogParser.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/RmSingleLineParser.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/stop-estimator.cmd
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/validator/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/SkylineStore.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/SingleLineParser.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/validator/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/api/Solver.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullRecurrenceIdException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/resourceestimator-config.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog5.txt
          • (edit) hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/ResourceSerDe.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ShutdownHook.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestResourceSkylineSerDe.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/validator/ParserValidator.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/api/TestResourceSkyline.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog6.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/service/TestResourceEstimatorService.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/NativeSingleLineParser.java
          • (add) hadoop-tools/hadoop-resourceestimator/pom.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullRLESparseResourceAllocationException.java
          • (edit) hadoop-tools/hadoop-tools-dist/pom.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/JobMetaData.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/start-estimator.sh
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/answer.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/log4j.properties
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/PredictionSkylineStore.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/preprocess/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestHistorySkylineSerDe.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/LogParserUtil.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/api/TestJobMetaData.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/package-info.java
          • (add) hadoop-assemblies/src/main/resources/assemblies/hadoop-resourceestimator.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/tpch_q12.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/data/resourceEstimatorService.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/start-estimator.cmd
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/skylinestore/impl/TestInMemoryStore.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/impl/InMemoryStore.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/RecurrenceId.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/HistorySkylineStore.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullPipelineIdException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/resourceEstimatorService.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/DuplicateRecurrenceIdException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/RLESparseResourceAllocationSerDe.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/ResourceEstimatorConfiguration.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ResourceEstimatorServer.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/LpSolver.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog7.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/resources/webapps/ResourceEstimatorServer/.gitignore
          • (edit) hadoop-project/src/site/site.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/impl/TestNativeParser.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/tinySample.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/api/package-info.java
          • (edit) LICENSE.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog2.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullResourceSkylineException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/stop-estimator.sh
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/nativeLog.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/impl/TestRmParser.java
          • (edit) hadoop-project/pom.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/ResourceSkyline.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog1.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/service/GuiceServletConfig.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/BaseLogParser.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog4.txt
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/ResourceEstimatorUtil.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/conf/resourceestimator-config.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/preprocess/SolverPreprocessor.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/skylinestore/impl/TestSkylineStore.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/InvalidInputException.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/site/markdown/ResourceEstimator.md
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/estimator.cmd
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/impl/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestResourceSerDe.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/package-info.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ResourceEstimatorService.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/BaseSolver.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/solver/impl/TestLpSolver.java
          • (add) hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml
          • (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog9.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #13138 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13138/ ) HADOOP-14840 . Tool to estimate resource requirements of an application (subru: rev 625039ef20e6011ab360131d70582a6e4bf2ec1d) (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/solver/impl/TestSolver.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/validator/SkylineStoreValidator.java (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/estimator.sh (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/RecurrenceIdNotFoundException.java (add) hadoop-tools/hadoop-resourceestimator/README.md (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/rmLog.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/SkylineStoreException.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/exceptions/DataFieldNotFoundException.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/SolverException.java (add) hadoop-tools/hadoop-resourceestimator/src/site/resources/css/site.css (edit) hadoop-tools/pom.xml (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/exception/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/InvalidSolverException.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/exceptions/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog8.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/EmptyResourceSkylineException.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog3.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/exception/ResourceEstimatorException.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/LogParser.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/RmSingleLineParser.java (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/stop-estimator.cmd (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/validator/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/SkylineStore.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/SingleLineParser.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/validator/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/api/Solver.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullRecurrenceIdException.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/resourceestimator-config.xml (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog5.txt (edit) hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/ResourceSerDe.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ShutdownHook.java (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestResourceSkylineSerDe.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/validator/ParserValidator.java (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/api/TestResourceSkyline.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog6.txt (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/service/TestResourceEstimatorService.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/NativeSingleLineParser.java (add) hadoop-tools/hadoop-resourceestimator/pom.xml (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullRLESparseResourceAllocationException.java (edit) hadoop-tools/hadoop-tools-dist/pom.xml (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/JobMetaData.java (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/start-estimator.sh (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/answer.txt (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/log4j.properties (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/PredictionSkylineStore.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/preprocess/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestHistorySkylineSerDe.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/LogParserUtil.java (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/api/TestJobMetaData.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/package-info.java (add) hadoop-assemblies/src/main/resources/assemblies/hadoop-resourceestimator.xml (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/tpch_q12.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/data/resourceEstimatorService.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/start-estimator.cmd (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/skylinestore/impl/TestInMemoryStore.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/impl/InMemoryStore.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/RecurrenceId.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/HistorySkylineStore.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullPipelineIdException.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/resourceEstimatorService.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/DuplicateRecurrenceIdException.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/RLESparseResourceAllocationSerDe.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/ResourceEstimatorConfiguration.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ResourceEstimatorServer.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/LpSolver.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog7.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/resources/webapps/ResourceEstimatorServer/.gitignore (edit) hadoop-project/src/site/site.xml (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/impl/TestNativeParser.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/tinySample.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/api/package-info.java (edit) LICENSE.txt (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog2.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullResourceSkylineException.java (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/stop-estimator.sh (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/nativeLog.txt (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/impl/TestRmParser.java (edit) hadoop-project/pom.xml (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/ResourceSkyline.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog1.txt (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/service/GuiceServletConfig.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/BaseLogParser.java (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog4.txt (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/ResourceEstimatorUtil.java (add) hadoop-tools/hadoop-resourceestimator/src/main/conf/resourceestimator-config.xml (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/preprocess/SolverPreprocessor.java (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/skylinestore/impl/TestSkylineStore.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/InvalidInputException.java (add) hadoop-tools/hadoop-resourceestimator/src/site/markdown/ResourceEstimator.md (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/estimator.cmd (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/impl/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestResourceSerDe.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/package-info.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ResourceEstimatorService.java (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/BaseSolver.java (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/solver/impl/TestLpSolver.java (add) hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog9.txt
          Hide
          subru Subru Krishnan added a comment -

          Thanks Rui Li for the contribution. Congrats on your first patch and way to go in starting with a big bang .

          Thanks to Sergiy Matusevych and Carlo Curino for the reviews.

          Show
          subru Subru Krishnan added a comment - Thanks Rui Li for the contribution. Congrats on your first patch and way to go in starting with a big bang . Thanks to Sergiy Matusevych and Carlo Curino for the reviews.
          Hide
          motus Sergiy Matusevych added a comment -

          Looks good. I have a few comments about the style (using more idiomatic Java, etc.), but I will fix the code and send out the patch later after this update gets merged into the main trunk.

          Show
          motus Sergiy Matusevych added a comment - Looks good. I have a few comments about the style (using more idiomatic Java, etc.), but I will fix the code and send out the patch later after this update gets merged into the main trunk.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 31 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 34s Maven dependency ordering for branch
          +1 mvninstall 14m 19s trunk passed
          +1 compile 15m 58s trunk passed
          +1 checkstyle 2m 16s trunk passed
          +1 mvnsite 10m 12s trunk passed
          +1 shadedclient 10m 20s branch has no errors when building and testing our client artifacts.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies hadoop-tools/hadoop-tools-dist hadoop-tools .
          +1 findbugs 0m 0s trunk passed
          +1 javadoc 6m 5s trunk passed
                Patch Compile Tests
          0 mvndep 0m 23s Maven dependency ordering for patch
          +1 mvninstall 28m 30s the patch passed
          +1 compile 24m 13s the patch passed
          +1 javac 24m 13s the patch passed
          +1 checkstyle 3m 57s the patch passed
          +1 mvnsite 22m 52s the patch passed
          +1 shellcheck 0m 13s There were no new shellcheck issues.
          +1 shelldocs 0m 13s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 28s The patch has no ill-formed XML file.
          +1 shadedclient 15m 53s patch has no errors when building and testing our client artifacts.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 2m 2s the patch passed
          +1 javadoc 12m 4s the patch passed
                Other Tests
          -1 unit 23m 45s root in the patch failed.
          +1 asflicense 1m 34s The patch does not generate ASF License warnings.
          220m 54s



          Reason Tests
          Failed junit tests hadoop.security.TestShellBasedUnixGroupsMapping
            hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:ca8ddc6
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893654/HADOOP-14840-v4.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle
          uname Linux 8eda6ae2391d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b90750c
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_131
          shellcheck v0.4.6
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13565/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13565/testReport/
          modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13565/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 31 new or modified test files.       trunk Compile Tests 0 mvndep 1m 34s Maven dependency ordering for branch +1 mvninstall 14m 19s trunk passed +1 compile 15m 58s trunk passed +1 checkstyle 2m 16s trunk passed +1 mvnsite 10m 12s trunk passed +1 shadedclient 10m 20s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies hadoop-tools/hadoop-tools-dist hadoop-tools . +1 findbugs 0m 0s trunk passed +1 javadoc 6m 5s trunk passed       Patch Compile Tests 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 28m 30s the patch passed +1 compile 24m 13s the patch passed +1 javac 24m 13s the patch passed +1 checkstyle 3m 57s the patch passed +1 mvnsite 22m 52s the patch passed +1 shellcheck 0m 13s There were no new shellcheck issues. +1 shelldocs 0m 13s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 28s The patch has no ill-formed XML file. +1 shadedclient 15m 53s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 2m 2s the patch passed +1 javadoc 12m 4s the patch passed       Other Tests -1 unit 23m 45s root in the patch failed. +1 asflicense 1m 34s The patch does not generate ASF License warnings. 220m 54s Reason Tests Failed junit tests hadoop.security.TestShellBasedUnixGroupsMapping   hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:ca8ddc6 JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893654/HADOOP-14840-v4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle uname Linux 8eda6ae2391d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b90750c maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 shellcheck v0.4.6 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13565/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13565/testReport/ modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13565/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Rui Li Rui Li added a comment -

          I keep the .gitignore file in hadoop-tools/hadoop-resourceestimator/src/main/resources/webapps/ResourceEstimatorServer/, because this folder cannot be empty, or else the HttpServer2 cannot be properly started.

          Show
          Rui Li Rui Li added a comment - I keep the .gitignore file in hadoop-tools/hadoop-resourceestimator/src/main/resources/webapps/ResourceEstimatorServer/, because this folder cannot be empty, or else the HttpServer2 cannot be properly started.
          Hide
          subru Subru Krishnan added a comment -

          Sergiy Matusevych, you can take a look at the latest uploaded patch (v3) and leave your comments directly in this jira. Thanks!

          Show
          subru Subru Krishnan added a comment - Sergiy Matusevych , you can take a look at the latest uploaded patch (v3) and leave your comments directly in this jira. Thanks!
          Hide
          motus Sergiy Matusevych added a comment -

          Subru Krishnan, Rui Li, I would love to take another look at the code - do we have it as a branch on github or somewhere where I can review it and leave comments or send updates?

          Show
          motus Sergiy Matusevych added a comment - Subru Krishnan , Rui Li , I would love to take another look at the code - do we have it as a branch on github or somewhere where I can review it and leave comments or send updates?
          Hide
          subru Subru Krishnan added a comment -

          Thanks Rui Li for working on this. A couple of nits on the documentation:

          • The README.md has couple of spurious lines -
             # ??? description and point to our .md file. 

            . You don't have a link to the main doc.

          • Please add description for config properties both in resourceestimator-config.xml and ResourceEstimatorConfiguration. You can simply copy from ResourceEstimator.md.
          • Remove the gitignore and logback.xml as they are not used anymore.

          Overall I am good with the patch as Sergiy Matusevych, Carlo Curino and myself have reviewed it internally. I'll wait for a couple of days to see if anyone else wants to take a look before committing.

          Show
          subru Subru Krishnan added a comment - Thanks Rui Li for working on this. A couple of nits on the documentation: The README.md has couple of spurious lines - # ??? description and point to our .md file. . You don't have a link to the main doc. Please add description for config properties both in resourceestimator-config.xml and ResourceEstimatorConfiguration . You can simply copy from ResourceEstimator.md . Remove the gitignore and logback.xml as they are not used anymore. Overall I am good with the patch as Sergiy Matusevych , Carlo Curino and myself have reviewed it internally. I'll wait for a couple of days to see if anyone else wants to take a look before committing.
          Hide
          Rui Li Rui Li added a comment -

          I can run the test code for hadoop resource estimator successfully in my local environment. Also, the test failures I got in the patch test are unrelated to my code changes.

          Show
          Rui Li Rui Li added a comment - I can run the test code for hadoop resource estimator successfully in my local environment. Also, the test failures I got in the patch test are unrelated to my code changes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 31 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 34s Maven dependency ordering for branch
          +1 mvninstall 14m 9s trunk passed
          +1 compile 13m 8s trunk passed
          +1 checkstyle 2m 13s trunk passed
          +1 mvnsite 10m 56s trunk passed
          +1 shadedclient 10m 20s branch has no errors when building and testing our client artifacts.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies hadoop-tools/hadoop-tools-dist hadoop-tools .
          +1 findbugs 0m 0s trunk passed
          +1 javadoc 5m 32s trunk passed
                Patch Compile Tests
          0 mvndep 0m 25s Maven dependency ordering for patch
          +1 mvninstall 17m 21s the patch passed
          +1 compile 17m 38s the patch passed
          +1 javac 17m 38s the patch passed
          +1 checkstyle 2m 37s the patch passed
          +1 mvnsite 12m 13s the patch passed
          +1 shellcheck 0m 12s There were no new shellcheck issues.
          +1 shelldocs 0m 10s There were no new shelldocs issues.
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 xml 0m 14s The patch has no ill-formed XML file.
          +1 shadedclient 11m 15s patch has no errors when building and testing our client artifacts.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 0m 58s the patch passed
          +1 javadoc 6m 17s the patch passed
                Other Tests
          -1 unit 19m 29s root in the patch failed.
          +1 asflicense 0m 54s The patch does not generate ASF License warnings.
          170m 13s



          Reason Tests
          Failed junit tests hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:ca8ddc6
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893202/HADOOP-14840-v3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle
          uname Linux 03497b9158f9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 248d9b6
          Default Java 1.8.0_131
          shellcheck v0.4.6
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13562/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13562/testReport/
          modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13562/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 31 new or modified test files.       trunk Compile Tests 0 mvndep 1m 34s Maven dependency ordering for branch +1 mvninstall 14m 9s trunk passed +1 compile 13m 8s trunk passed +1 checkstyle 2m 13s trunk passed +1 mvnsite 10m 56s trunk passed +1 shadedclient 10m 20s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies hadoop-tools/hadoop-tools-dist hadoop-tools . +1 findbugs 0m 0s trunk passed +1 javadoc 5m 32s trunk passed       Patch Compile Tests 0 mvndep 0m 25s Maven dependency ordering for patch +1 mvninstall 17m 21s the patch passed +1 compile 17m 38s the patch passed +1 javac 17m 38s the patch passed +1 checkstyle 2m 37s the patch passed +1 mvnsite 12m 13s the patch passed +1 shellcheck 0m 12s There were no new shellcheck issues. +1 shelldocs 0m 10s There were no new shelldocs issues. +1 whitespace 0m 1s The patch has no whitespace issues. +1 xml 0m 14s The patch has no ill-formed XML file. +1 shadedclient 11m 15s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 0m 58s the patch passed +1 javadoc 6m 17s the patch passed       Other Tests -1 unit 19m 29s root in the patch failed. +1 asflicense 0m 54s The patch does not generate ASF License warnings. 170m 13s Reason Tests Failed junit tests hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem Subsystem Report/Notes Docker Image:yetus/hadoop:ca8ddc6 JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893202/HADOOP-14840-v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle uname Linux 03497b9158f9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 248d9b6 Default Java 1.8.0_131 shellcheck v0.4.6 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13562/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13562/testReport/ modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13562/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 10m 5s Docker mode activated.
                Prechecks
          0 findbugs 0m 0s Findbugs executables are not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 31 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 35s Maven dependency ordering for branch
          +1 mvninstall 16m 12s trunk passed
          +1 compile 14m 57s trunk passed
          +1 checkstyle 2m 4s trunk passed
          +1 mvnsite 9m 10s trunk passed
          +1 shadedclient 9m 57s branch has no errors when building and testing our client artifacts.
          +1 javadoc 4m 50s trunk passed
                Patch Compile Tests
          0 mvndep 0m 25s Maven dependency ordering for patch
          +1 mvninstall 16m 24s the patch passed
          +1 compile 11m 0s the patch passed
          +1 javac 11m 0s the patch passed
          -0 checkstyle 2m 10s root: The patch generated 27 new + 0 unchanged - 0 fixed = 27 total (was 0)
          +1 mvnsite 9m 10s the patch passed
          +1 shellcheck 0m 11s There were no new shellcheck issues.
          +1 shelldocs 0m 10s There were no new shelldocs issues.
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 xml 0m 11s The patch has no ill-formed XML file.
          +1 shadedclient 10m 11s patch has no errors when building and testing our client artifacts.
          +1 javadoc 4m 48s the patch passed
                Other Tests
          -1 unit 12m 57s root in the patch failed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          159m 15s



          Reason Tests
          Failed junit tests hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:ca8ddc6
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893138/HADOOP-14840-v2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle
          uname Linux 1de89c74f31a 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7b4b018
          Default Java 1.8.0_131
          shellcheck v0.4.6
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13557/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13557/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13557/testReport/
          modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13557/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 5s Docker mode activated.       Prechecks 0 findbugs 0m 0s Findbugs executables are not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 31 new or modified test files.       trunk Compile Tests 0 mvndep 1m 35s Maven dependency ordering for branch +1 mvninstall 16m 12s trunk passed +1 compile 14m 57s trunk passed +1 checkstyle 2m 4s trunk passed +1 mvnsite 9m 10s trunk passed +1 shadedclient 9m 57s branch has no errors when building and testing our client artifacts. +1 javadoc 4m 50s trunk passed       Patch Compile Tests 0 mvndep 0m 25s Maven dependency ordering for patch +1 mvninstall 16m 24s the patch passed +1 compile 11m 0s the patch passed +1 javac 11m 0s the patch passed -0 checkstyle 2m 10s root: The patch generated 27 new + 0 unchanged - 0 fixed = 27 total (was 0) +1 mvnsite 9m 10s the patch passed +1 shellcheck 0m 11s There were no new shellcheck issues. +1 shelldocs 0m 10s There were no new shelldocs issues. +1 whitespace 0m 1s The patch has no whitespace issues. +1 xml 0m 11s The patch has no ill-formed XML file. +1 shadedclient 10m 11s patch has no errors when building and testing our client artifacts. +1 javadoc 4m 48s the patch passed       Other Tests -1 unit 12m 57s root in the patch failed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 159m 15s Reason Tests Failed junit tests hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem Subsystem Report/Notes Docker Image:yetus/hadoop:ca8ddc6 JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893138/HADOOP-14840-v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle uname Linux 1de89c74f31a 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7b4b018 Default Java 1.8.0_131 shellcheck v0.4.6 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13557/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13557/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13557/testReport/ modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13557/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 13s Docker failed to build yetus/hadoop:0de40f0.



          Subsystem Report/Notes
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893138/HADOOP-14840-v2.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13556/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 13s Docker failed to build yetus/hadoop:0de40f0. Subsystem Report/Notes JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893138/HADOOP-14840-v2.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13556/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 11s Docker failed to build yetus/hadoop:0de40f0.



          Subsystem Report/Notes
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893097/HADOOP-14840-v1.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13555/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 11s Docker failed to build yetus/hadoop:0de40f0. Subsystem Report/Notes JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893097/HADOOP-14840-v1.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13555/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 12s Docker failed to build yetus/hadoop:0de40f0.



          Subsystem Report/Notes
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893098/HADOOP-14840-v2.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13554/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 12s Docker failed to build yetus/hadoop:0de40f0. Subsystem Report/Notes JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893098/HADOOP-14840-v2.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13554/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Rui Li, please do not remove older patches as the provenance will be lost. That's the reason we use version numbers, so reattach the v1 patch and then the updated patch as v2. Thanks.

          Show
          subru Subru Krishnan added a comment - Rui Li , please do not remove older patches as the provenance will be lost. That's the reason we use version numbers, so reattach the v1 patch and then the updated patch as v2. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 10s Docker failed to build yetus/hadoop:0de40f0.



          Subsystem Report/Notes
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892980/HADOOP-14840-v1.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13551/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 10s Docker failed to build yetus/hadoop:0de40f0. Subsystem Report/Notes JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892980/HADOOP-14840-v1.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13551/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 30 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 15m 15s trunk passed
          +1 compile 15m 33s trunk passed
          +1 checkstyle 2m 19s trunk passed
          +1 mvnsite 11m 16s trunk passed
          +1 shadedclient 10m 28s branch has no errors when building and testing our client artifacts.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies hadoop-tools/hadoop-tools-dist hadoop-tools .
          +1 findbugs 0m 0s trunk passed
          +1 javadoc 5m 34s trunk passed
                Patch Compile Tests
          0 mvndep 0m 25s Maven dependency ordering for patch
          +1 mvninstall 18m 22s the patch passed
          +1 compile 12m 58s the patch passed
          +1 javac 12m 58s the patch passed
          -0 checkstyle 2m 21s root: The patch generated 130 new + 0 unchanged - 0 fixed = 130 total (was 0)
          +1 mvnsite 10m 51s the patch passed
          -1 shellcheck 0m 0s The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)
          -0 shelldocs 0m 9s The patch generated 6 new + 104 unchanged - 0 fixed = 110 total (was 104)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 15s The patch has no ill-formed XML file.
          +1 shadedclient 10m 51s patch has no errors when building and testing our client artifacts.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 0m 45s the patch passed
          +1 javadoc 5m 11s the patch passed
                Other Tests
          -1 unit 14m 38s root in the patch failed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          160m 22s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0de40f0
          JIRA Issue HADOOP-14840
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892740/HADOOP-14840-v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle
          uname Linux 74f950b1657f 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2523e1c
          Default Java 1.8.0_144
          shellcheck v0.4.6
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/artifact/patchprocess/diff-checkstyle-root.txt
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/artifact/patchprocess/diff-patch-shellcheck.txt
          shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/artifact/patchprocess/diff-patch-shelldocs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/testReport/
          modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 30 new or modified test files.       trunk Compile Tests 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 15m 15s trunk passed +1 compile 15m 33s trunk passed +1 checkstyle 2m 19s trunk passed +1 mvnsite 11m 16s trunk passed +1 shadedclient 10m 28s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies hadoop-tools/hadoop-tools-dist hadoop-tools . +1 findbugs 0m 0s trunk passed +1 javadoc 5m 34s trunk passed       Patch Compile Tests 0 mvndep 0m 25s Maven dependency ordering for patch +1 mvninstall 18m 22s the patch passed +1 compile 12m 58s the patch passed +1 javac 12m 58s the patch passed -0 checkstyle 2m 21s root: The patch generated 130 new + 0 unchanged - 0 fixed = 130 total (was 0) +1 mvnsite 10m 51s the patch passed -1 shellcheck 0m 0s The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) -0 shelldocs 0m 9s The patch generated 6 new + 104 unchanged - 0 fixed = 110 total (was 104) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 15s The patch has no ill-formed XML file. +1 shadedclient 10m 51s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-assemblies . hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 0m 45s the patch passed +1 javadoc 5m 11s the patch passed       Other Tests -1 unit 14m 38s root in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 160m 22s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:0de40f0 JIRA Issue HADOOP-14840 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892740/HADOOP-14840-v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml shellcheck shelldocs findbugs checkstyle uname Linux 74f950b1657f 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2523e1c Default Java 1.8.0_144 shellcheck v0.4.6 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/artifact/patchprocess/diff-checkstyle-root.txt shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/artifact/patchprocess/diff-patch-shellcheck.txt shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/artifact/patchprocess/diff-patch-shelldocs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/testReport/ modules C: hadoop-project hadoop-assemblies hadoop-tools/hadoop-resourceestimator . hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13541/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Attaching design doc

          Show
          subru Subru Krishnan added a comment - Attaching design doc
          Hide
          vrushalic Vrushali C added a comment -

          Thanks Subru for the response!

          No, we don't need to break it up into smaller patches. My question was more out of curiosity and was wondering if this was still at an initial stage and if so, would there be sub tasks. Good to hear that you already have something running internally and want to contribute it back to the community. Will look forward to the design doc with the perspective of understanding it for future use cases for TSv2. Will also look through the paper.

          thanks!

          Show
          vrushalic Vrushali C added a comment - Thanks Subru for the response! No, we don't need to break it up into smaller patches. My question was more out of curiosity and was wondering if this was still at an initial stage and if so, would there be sub tasks. Good to hear that you already have something running internally and want to contribute it back to the community. Will look forward to the design doc with the perspective of understanding it for future use cases for TSv2. Will also look through the paper. thanks!
          Hide
          subru Subru Krishnan added a comment -

          Vrushali C, thanks for your comments, please find my response inline.

          This sounds interesting. Will there be subtasks under this?

          We have developed this tool and are using it internally. So it has already been through rigorous reviews etc so was just planning on a single uber patch to save time/efficiency. We can break it up if really required.

          Similar estimations for MR jobs like reducer estimation & memory estimation are done based on past runs using a tool called hRaven (https://github.com/twitter/hraven)

          We are aware of hRaven. In fact, we have a similar work for Hive and Tez called PerfOrator. With the ever growing frameworks (and optimizations/updates to existing ones), we decided to follow a different approach here:

          • Be framework agnostic, i.e. we model only resources over time and so are not limited to map or reduce (or Tez or Reef). We are essentially emulating the YARN model natively.
          • Work purely based on history (can be direct hook to hRaven). We have developed a Linear Programming model to estimate the resources required for the job's future execution.
          • Optionally reserve the resources using YARN's ReservationSystem.

          The gory details are available in the Morpheus paper.

          I would think of this as a natural extension to Timeline Service v2. Do you think you might want to make use of TSv2 in any way for this?

          We can't agree more . The initial version we have works on YARN RM logs only because that's what is widely available (in prod deployments). We have designed it with the clear intent to integrate with TSv2 (or even hRaven) in future, with the community help, of course.

          We will upload a design doc shortly.

          Show
          subru Subru Krishnan added a comment - Vrushali C , thanks for your comments, please find my response inline. This sounds interesting. Will there be subtasks under this? We have developed this tool and are using it internally. So it has already been through rigorous reviews etc so was just planning on a single uber patch to save time/efficiency. We can break it up if really required. Similar estimations for MR jobs like reducer estimation & memory estimation are done based on past runs using a tool called hRaven ( https://github.com/twitter/hraven ) We are aware of hRaven. In fact, we have a similar work for Hive and Tez called PerfOrator . With the ever growing frameworks (and optimizations/updates to existing ones), we decided to follow a different approach here: Be framework agnostic, i.e. we model only resources over time and so are not limited to map or reduce (or Tez or Reef). We are essentially emulating the YARN model natively. Work purely based on history (can be direct hook to hRaven). We have developed a Linear Programming model to estimate the resources required for the job's future execution. Optionally reserve the resources using YARN's ReservationSystem. The gory details are available in the Morpheus paper . I would think of this as a natural extension to Timeline Service v2. Do you think you might want to make use of TSv2 in any way for this? We can't agree more . The initial version we have works on YARN RM logs only because that's what is widely available (in prod deployments). We have designed it with the clear intent to integrate with TSv2 (or even hRaven) in future, with the community help, of course. We will upload a design doc shortly.
          Hide
          vrushalic Vrushali C added a comment -

          This sounds interesting. Will there be subtasks under this?

          Similar estimations for MR jobs like reducer estimation & memory estimation are done based on past runs using a tool called hRaven (https://github.com/twitter/hraven). I would think of this as a natural extension to Timeline Service v2. Do you think you might want to make use of TSv2 in any way for this?

          Show
          vrushalic Vrushali C added a comment - This sounds interesting. Will there be subtasks under this? Similar estimations for MR jobs like reducer estimation & memory estimation are done based on past runs using a tool called hRaven ( https://github.com/twitter/hraven ). I would think of this as a natural extension to Timeline Service v2. Do you think you might want to make use of TSv2 in any way for this?

            People

            • Assignee:
              Rui Li Rui Li
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development