Details

    • Hadoop Flags:
      Reviewed

      Description

      The timeline service v2 code base on yarn now uses hbase 1.2.4 after YARN-5976.

      With this version of hbase, system classes (starting with org.apache.hadoop) can be loaded as table level coprocessors. Hence we should update the timeline service coprocessor to be a dynamically loaded one instead of static loading.

      This involves code changes as well as documentation updates for deployment.

      1. YARN-6094-YARN-5355.004.patch
        54 kB
        Vrushali C
      2. YARN-6094-YARN-5355.003.patch
        53 kB
        Vrushali C
      3. YARN-6094-YARN-5355.002.patch
        96 kB
        Vrushali C
      4. YARN-6094-YARN-5355.001.patch
        61 kB
        Vrushali C
      5. YARN-6094.001.patch
        61 kB
        Vrushali C

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12271 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12271/)
        YARN-6094. Update the coprocessor to be a dynamically loaded one. (varunsaxena: rev 580d884913c200f6c6cae475d8c183a4b61ff710)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestHBaseStorageFlowActivity.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestHBaseStorageFlowRun.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorageEntities.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTable.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorageSchema.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunCoprocessor.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/DataGeneratorForTest.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestHBaseStorageFlowRunCompaction.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestFlowDataGenerator.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWebServicesHBaseStorage.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorageApps.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12271 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12271/ ) YARN-6094 . Update the coprocessor to be a dynamically loaded one. (varunsaxena: rev 580d884913c200f6c6cae475d8c183a4b61ff710) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestHBaseStorageFlowActivity.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestHBaseStorageFlowRun.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorageEntities.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTable.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorageSchema.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunCoprocessor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/DataGeneratorForTest.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestHBaseStorageFlowRunCompaction.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestFlowDataGenerator.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWebServicesHBaseStorage.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorageApps.java
        Hide
        sjlee0 Sangjin Lee added a comment -

        Committed the change to YARN-5355 and YARN-5355-branch-2. Thanks Vrushali C for your contribution!

        Show
        sjlee0 Sangjin Lee added a comment - Committed the change to YARN-5355 and YARN-5355 -branch-2. Thanks Vrushali C for your contribution!
        Hide
        sjlee0 Sangjin Lee added a comment -

        Sounds good. Committing shortly.

        Show
        sjlee0 Sangjin Lee added a comment - Sounds good. Committing shortly.
        Hide
        vrushalic Vrushali C added a comment -

        Yes tags require hfile3 format. HBase 1.0 onwards hfile3 is the default I believe https://issues.apache.org/jira/browse/HBASE-9801.

        But yes we can add to the documentation that timeline service uses cell tags and hence needs hfile3.

        Show
        vrushalic Vrushali C added a comment - Yes tags require hfile3 format. HBase 1.0 onwards hfile3 is the default I believe https://issues.apache.org/jira/browse/HBASE-9801 . But yes we can add to the documentation that timeline service uses cell tags and hence needs hfile3.
        Hide
        esteban Esteban Gutierrez added a comment -

        I don't thin thats necessary, our HBase hfile.format.version has been bumped since HBase 0.98 to use HFileV3 by default. However we do still carry some tests that programmatically pass that value before stating the HBase minicluster. Some distributions might have opted to ship revert to the old HFile version due some compat requirements but as long is documented that the Timeline Service needs to use tags and is expected that hfile.format.version is using the default of 3 I don't see any problem.

        Show
        esteban Esteban Gutierrez added a comment - I don't thin thats necessary, our HBase hfile.format.version has been bumped since HBase 0.98 to use HFileV3 by default. However we do still carry some tests that programmatically pass that value before stating the HBase minicluster. Some distributions might have opted to ship revert to the old HFile version due some compat requirements but as long is documented that the Timeline Service needs to use tags and is expected that hfile.format.version is using the default of 3 I don't see any problem.
        Hide
        sjlee0 Sangjin Lee added a comment -

        I see that hfile.format.version = 3 is actually being used in multiple places in the unit tests. But it also appears that the default value in hbase-default.xml is 3. It also states

          <property>
              <name>hfile.format.version</name>
              <value>3</value>
              <description>The HFile format version to use for new files.
              Version 3 adds support for tags in hfiles (See http://hbase.apache.org/book.html#hbase.tags).
              Distributed Log Replay requires that tags are enabled. Also see the configuration
              'hbase.replication.rpc.codec'.
              </description>
          </property>
        

        It seems to me that the timeline service requires this to be 3 because we're relying on tags, right? Then should we add this requirement to the documentation? If needed, I suppose doing this as part of the documentation update JIRA is an option.

        Show
        sjlee0 Sangjin Lee added a comment - I see that hfile.format.version = 3 is actually being used in multiple places in the unit tests. But it also appears that the default value in hbase-default.xml is 3. It also states <property> <name>hfile.format.version</name> <value>3</value> <description>The HFile format version to use for new files. Version 3 adds support for tags in hfiles (See http://hbase.apache.org/book.html#hbase.tags). Distributed Log Replay requires that tags are enabled. Also see the configuration 'hbase.replication.rpc.codec'. </description> </property> It seems to me that the timeline service requires this to be 3 because we're relying on tags, right? Then should we add this requirement to the documentation? If needed, I suppose doing this as part of the documentation update JIRA is an option.
        Hide
        sjlee0 Sangjin Lee added a comment -

        I went over the latest patch, and noticed one thing (sorry should have seen them earlier).

        In TestHBaseTimelineStorageSchema I see the following line has been added:

            conf.setInt("hfile.format.version", 3);
        

        What does this signify? Why is this change needed?

        Show
        sjlee0 Sangjin Lee added a comment - I went over the latest patch, and noticed one thing (sorry should have seen them earlier). In TestHBaseTimelineStorageSchema I see the following line has been added: conf.setInt( "hfile.format.version" , 3); What does this signify? Why is this change needed?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 2m 42s Maven dependency ordering for branch
        +1 mvninstall 8m 15s YARN-5355 passed
        +1 compile 2m 20s YARN-5355 passed
        +1 checkstyle 0m 41s YARN-5355 passed
        +1 mvnsite 1m 52s YARN-5355 passed
        +1 mvneclipse 1m 3s YARN-5355 passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 2m 24s YARN-5355 passed
        +1 javadoc 1m 19s YARN-5355 passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 2m 15s the patch passed
        +1 javac 2m 15s the patch passed
        +1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217)
        +1 mvnsite 1m 44s the patch passed
        +1 mvneclipse 0m 57s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 2m 51s the patch passed
        +1 javadoc 1m 15s the patch passed
        +1 unit 0m 26s hadoop-yarn-api in the patch passed.
        +1 unit 2m 17s hadoop-yarn-common in the patch passed.
        +1 unit 0m 44s hadoop-yarn-server-timelineservice in the patch passed.
        +1 unit 4m 49s hadoop-yarn-server-timelineservice-hbase-tests in the patch passed.
        +1 unit 0m 9s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        48m 57s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-6094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848384/YARN-6094-YARN-5355.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 391f5a2000dc 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5355 / c9246f6
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14709/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14709/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 2m 42s Maven dependency ordering for branch +1 mvninstall 8m 15s YARN-5355 passed +1 compile 2m 20s YARN-5355 passed +1 checkstyle 0m 41s YARN-5355 passed +1 mvnsite 1m 52s YARN-5355 passed +1 mvneclipse 1m 3s YARN-5355 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 2m 24s YARN-5355 passed +1 javadoc 1m 19s YARN-5355 passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 2m 15s the patch passed +1 javac 2m 15s the patch passed +1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217) +1 mvnsite 1m 44s the patch passed +1 mvneclipse 0m 57s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 2m 51s the patch passed +1 javadoc 1m 15s the patch passed +1 unit 0m 26s hadoop-yarn-api in the patch passed. +1 unit 2m 17s hadoop-yarn-common in the patch passed. +1 unit 0m 44s hadoop-yarn-server-timelineservice in the patch passed. +1 unit 4m 49s hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. +1 unit 0m 9s hadoop-yarn-site in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 48m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-6094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848384/YARN-6094-YARN-5355.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 391f5a2000dc 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5355 / c9246f6 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14709/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14709/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vrushalic Vrushali C added a comment -

        Uploading v004 that adds the default settings to yarn-default.xml and moves the default value setting to YarnConfiguration as per Sangjin Lee's recommendation

        Show
        vrushalic Vrushali C added a comment - Uploading v004 that adds the default settings to yarn-default.xml and moves the default value setting to YarnConfiguration as per Sangjin Lee 's recommendation
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 0m 18s Maven dependency ordering for branch
        +1 mvninstall 7m 51s YARN-5355 passed
        +1 compile 2m 36s YARN-5355 passed
        +1 checkstyle 0m 40s YARN-5355 passed
        +1 mvnsite 1m 24s YARN-5355 passed
        +1 mvneclipse 0m 55s YARN-5355 passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 1m 38s YARN-5355 passed
        +1 javadoc 0m 53s YARN-5355 passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 14s the patch passed
        +1 compile 2m 41s the patch passed
        +1 javac 2m 41s the patch passed
        +1 checkstyle 0m 44s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217)
        +1 mvnsite 1m 23s the patch passed
        +1 mvneclipse 0m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 2m 3s the patch passed
        +1 javadoc 0m 52s the patch passed
        +1 unit 0m 28s hadoop-yarn-api in the patch passed.
        +1 unit 0m 51s hadoop-yarn-server-timelineservice in the patch passed.
        +1 unit 5m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch passed.
        +1 unit 0m 9s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        41m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-6094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848157/YARN-6094-YARN-5355.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 222a6183a84b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5355 / c9246f6
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14702/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14702/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 51s YARN-5355 passed +1 compile 2m 36s YARN-5355 passed +1 checkstyle 0m 40s YARN-5355 passed +1 mvnsite 1m 24s YARN-5355 passed +1 mvneclipse 0m 55s YARN-5355 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 38s YARN-5355 passed +1 javadoc 0m 53s YARN-5355 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 2m 41s the patch passed +1 javac 2m 41s the patch passed +1 checkstyle 0m 44s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 2m 3s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 0m 28s hadoop-yarn-api in the patch passed. +1 unit 0m 51s hadoop-yarn-server-timelineservice in the patch passed. +1 unit 5m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. +1 unit 0m 9s hadoop-yarn-site in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 41m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-6094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848157/YARN-6094-YARN-5355.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 222a6183a84b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5355 / c9246f6 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14702/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14702/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sjlee0 Sangjin Lee added a comment -

        The build failures are caused by an existing issue on the branch.

        The latest patch looks good for the most part. A couple of minor things:

        • We should add the new config property along with its default value in yarn-default.xml
        • We should declare the default value in YarnConfiguration.java next to the key, instead of FlowRunTable.java
        Show
        sjlee0 Sangjin Lee added a comment - The build failures are caused by an existing issue on the branch. The latest patch looks good for the most part. A couple of minor things: We should add the new config property along with its default value in yarn-default.xml We should declare the default value in YarnConfiguration.java next to the key, instead of FlowRunTable.java
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 0m 41s Maven dependency ordering for branch
        -1 mvninstall 3m 40s root in YARN-5355 failed.
        -1 compile 1m 18s hadoop-yarn in YARN-5355 failed.
        +1 checkstyle 0m 39s YARN-5355 passed
        -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice in YARN-5355 failed.
        -1 mvnsite 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in YARN-5355 failed.
        +1 mvneclipse 0m 43s YARN-5355 passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 13s hadoop-yarn-server-timelineservice in YARN-5355 failed.
        +1 javadoc 0m 47s YARN-5355 passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        -1 mvninstall 0m 13s hadoop-yarn-server-timelineservice in the patch failed.
        -1 mvninstall 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        -1 compile 1m 18s hadoop-yarn in the patch failed.
        -1 javac 1m 18s hadoop-yarn in the patch failed.
        +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217)
        -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice in the patch failed.
        -1 mvnsite 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        +1 mvneclipse 0m 43s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 13s hadoop-yarn-server-timelineservice in the patch failed.
        +1 javadoc 0m 47s the patch passed
        +1 unit 0m 26s hadoop-yarn-api in the patch passed.
        -1 unit 0m 15s hadoop-yarn-server-timelineservice in the patch failed.
        -1 unit 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        +1 unit 0m 8s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        26m 9s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-6094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848157/YARN-6094-YARN-5355.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6d02138961d3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5355 / 4b1ba4e
        Default Java 1.8.0_111
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-mvninstall-root.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14696/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14696/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 41s Maven dependency ordering for branch -1 mvninstall 3m 40s root in YARN-5355 failed. -1 compile 1m 18s hadoop-yarn in YARN-5355 failed. +1 checkstyle 0m 39s YARN-5355 passed -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice in YARN-5355 failed. -1 mvnsite 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in YARN-5355 failed. +1 mvneclipse 0m 43s YARN-5355 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 13s hadoop-yarn-server-timelineservice in YARN-5355 failed. +1 javadoc 0m 47s YARN-5355 passed 0 mvndep 0m 9s Maven dependency ordering for patch -1 mvninstall 0m 13s hadoop-yarn-server-timelineservice in the patch failed. -1 mvninstall 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. -1 compile 1m 18s hadoop-yarn in the patch failed. -1 javac 1m 18s hadoop-yarn in the patch failed. +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217) -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice in the patch failed. -1 mvnsite 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 13s hadoop-yarn-server-timelineservice in the patch failed. +1 javadoc 0m 47s the patch passed +1 unit 0m 26s hadoop-yarn-api in the patch passed. -1 unit 0m 15s hadoop-yarn-server-timelineservice in the patch failed. -1 unit 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. +1 unit 0m 8s hadoop-yarn-site in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 26m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-6094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848157/YARN-6094-YARN-5355.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6d02138961d3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5355 / 4b1ba4e Default Java 1.8.0_111 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14696/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14696/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14696/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 0m 51s Maven dependency ordering for branch
        -1 mvninstall 3m 52s root in YARN-5355 failed.
        -1 compile 1m 18s hadoop-yarn in YARN-5355 failed.
        +1 checkstyle 0m 39s YARN-5355 passed
        -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice in YARN-5355 failed.
        -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice-hbase-tests in YARN-5355 failed.
        +1 mvneclipse 0m 44s YARN-5355 passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 14s hadoop-yarn-server-timelineservice in YARN-5355 failed.
        +1 javadoc 0m 48s YARN-5355 passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        -1 mvninstall 0m 13s hadoop-yarn-server-timelineservice in the patch failed.
        -1 mvninstall 0m 13s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        -1 compile 1m 18s hadoop-yarn in the patch failed.
        -1 javac 1m 18s hadoop-yarn in the patch failed.
        +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217)
        -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice in the patch failed.
        -1 mvnsite 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        +1 mvneclipse 0m 46s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 13s hadoop-yarn-server-timelineservice in the patch failed.
        +1 javadoc 0m 47s the patch passed
        +1 unit 0m 26s hadoop-yarn-api in the patch passed.
        -1 unit 0m 13s hadoop-yarn-server-timelineservice in the patch failed.
        -1 unit 0m 16s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        +1 unit 0m 8s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        26m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-6094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848157/YARN-6094-YARN-5355.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b44d6088890e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5355 / 4b1ba4e
        Default Java 1.8.0_111
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-mvninstall-root.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14694/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14694/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 51s Maven dependency ordering for branch -1 mvninstall 3m 52s root in YARN-5355 failed. -1 compile 1m 18s hadoop-yarn in YARN-5355 failed. +1 checkstyle 0m 39s YARN-5355 passed -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice in YARN-5355 failed. -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice-hbase-tests in YARN-5355 failed. +1 mvneclipse 0m 44s YARN-5355 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 14s hadoop-yarn-server-timelineservice in YARN-5355 failed. +1 javadoc 0m 48s YARN-5355 passed 0 mvndep 0m 10s Maven dependency ordering for patch -1 mvninstall 0m 13s hadoop-yarn-server-timelineservice in the patch failed. -1 mvninstall 0m 13s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. -1 compile 1m 18s hadoop-yarn in the patch failed. -1 javac 1m 18s hadoop-yarn in the patch failed. +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217) -1 mvnsite 0m 15s hadoop-yarn-server-timelineservice in the patch failed. -1 mvnsite 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 13s hadoop-yarn-server-timelineservice in the patch failed. +1 javadoc 0m 47s the patch passed +1 unit 0m 26s hadoop-yarn-api in the patch passed. -1 unit 0m 13s hadoop-yarn-server-timelineservice in the patch failed. -1 unit 0m 16s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. +1 unit 0m 8s hadoop-yarn-site in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 26m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-6094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848157/YARN-6094-YARN-5355.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b44d6088890e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5355 / 4b1ba4e Default Java 1.8.0_111 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14694/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14694/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vrushalic Vrushali C added a comment - - edited

        Thanks Sangjin Lee for the review and discussion. I thought over this a bit and looked into the hbase code as well and it appears that if we pass in null as the hdfs location, then it will load the table coprocessor from classpath. So it became much easier to do this for the unit tests.

        Also, I moved the test files back into the earlier places since it makes things easier to review and follow.

        Uploading patch 003.

        Show
        vrushalic Vrushali C added a comment - - edited Thanks Sangjin Lee for the review and discussion. I thought over this a bit and looked into the hbase code as well and it appears that if we pass in null as the hdfs location, then it will load the table coprocessor from classpath. So it became much easier to do this for the unit tests. Also, I moved the test files back into the earlier places since it makes things easier to review and follow. Uploading patch 003.
        Hide
        sjlee0 Sangjin Lee added a comment -

        Yes. I wanted to load the coprocessor jar from classpath but I am not certain how to do that in the unit test on the minicluster. We can discuss further on this.

        We spoke offline on this. There are more reliable ways to get to the right enclosing jar using the Class and ClassLoader, but a larger issue is that we cannot expect that the timeline service dependency is always in the form of a jar file when these unit tests run. If we want to stick with a dynamically loaded coprocessor, we need to create the jar on the fly. Otherwise, it might be acceptable to use static loading for most of the unit tests.

        Yes, dynamic loading API requires the coprocessor jar to be present on HDFS. The Region Server does not need a restart when dynamic loading is used.

        Thanks for the confirmation.

        These two classes DataGeneratorForTest and TestFlowDataGenerator are common to most of the unit tests and they should have been put in a common package but for some reason, we did not do that earlier. This move feels correct to me.
        Regarding the diffs, I understand the changes are not easily understandable since the files moved. I think I can upload a diff of the two files to make it easier for review, let me know.

        I am +1 if the move makes sense. By the way, there is a way to detect renames to generate a smaller patch. Add the renamed files (both deleted files and added files) to git index or even commit it locally, and then use git diff -M90% or even lower percentage to have git detect the rename.

        Show
        sjlee0 Sangjin Lee added a comment - Yes. I wanted to load the coprocessor jar from classpath but I am not certain how to do that in the unit test on the minicluster. We can discuss further on this. We spoke offline on this. There are more reliable ways to get to the right enclosing jar using the Class and ClassLoader , but a larger issue is that we cannot expect that the timeline service dependency is always in the form of a jar file when these unit tests run. If we want to stick with a dynamically loaded coprocessor, we need to create the jar on the fly. Otherwise, it might be acceptable to use static loading for most of the unit tests. Yes, dynamic loading API requires the coprocessor jar to be present on HDFS. The Region Server does not need a restart when dynamic loading is used. Thanks for the confirmation. These two classes DataGeneratorForTest and TestFlowDataGenerator are common to most of the unit tests and they should have been put in a common package but for some reason, we did not do that earlier. This move feels correct to me. Regarding the diffs, I understand the changes are not easily understandable since the files moved. I think I can upload a diff of the two files to make it easier for review, let me know. I am +1 if the move makes sense. By the way, there is a way to detect renames to generate a smaller patch. Add the renamed files (both deleted files and added files) to git index or even commit it locally, and then use git diff -M90% or even lower percentage to have git detect the rename.
        Hide
        vrushalic Vrushali C added a comment -

        Thanks Sangjin Lee for the review! Here are some answers:

        We need a different way to get at the location of the jar (coming from the maven local cache).

        Yes. I wanted to load the coprocessor jar from classpath but I am not certain how to do that in the unit test on the minicluster. We can discuss further on this.

        Question: I see we're making a change to upload and use the jar that has the co-processor from hdfs, rather than adding it to the hbase install. Is that change needed for the dynamic co-processor?

        Yes, dynamic loading API requires the coprocessor jar to be present on HDFS. The Region Server does not need a restart when dynamic loading is used.

        What is the benefit of doing that over using it out of the local install?

        The local install would mean a restart of the region server but this is what is called as static loading. When coprocessors are loaded this way, they are turned on for all tables. We don't want that. We would like the coprocessor to be enabled for only the flow run table, hence dynamic loading is the way to go.

        I always wanted to do dynamic loading but we could not use it since the previous version of hbase was not allowing system classes to be loaded as dynamically loaded coprocessors. So it's not a new thought, it was always the intention but was not possible due to hbase version dependency.

        maybe I'm bit on the paranoid side, but checking whether it is operating on the right table seems like a reasonable sanity check nonetheless; do we need to remove this check?

        Keeping the check won't break anything. But it will slow it down. Since this is a table level coprocessor, the check is unnecessary and at high loads, the performance would likely be affected.

        Do we need to move the package for this and TestFlowDataGenerator? I'm curious as to why that move is necessary. The diffs are now deleting the old files and creating the new files so it's not so easy to see what has changed.

        These two classes DataGeneratorForTest and TestFlowDataGenerator are common to most of the unit tests and they should have been put in a common package but for some reason, we did not do that earlier. This move feels correct to me.
        Regarding the diffs, I understand the changes are not easily understandable since the files moved. I think I can upload a diff of the two files to make it easier for review, let me know.

        Show
        vrushalic Vrushali C added a comment - Thanks Sangjin Lee for the review! Here are some answers: We need a different way to get at the location of the jar (coming from the maven local cache). Yes. I wanted to load the coprocessor jar from classpath but I am not certain how to do that in the unit test on the minicluster. We can discuss further on this. Question: I see we're making a change to upload and use the jar that has the co-processor from hdfs, rather than adding it to the hbase install. Is that change needed for the dynamic co-processor? Yes, dynamic loading API requires the coprocessor jar to be present on HDFS. The Region Server does not need a restart when dynamic loading is used. What is the benefit of doing that over using it out of the local install? The local install would mean a restart of the region server but this is what is called as static loading. When coprocessors are loaded this way, they are turned on for all tables. We don't want that. We would like the coprocessor to be enabled for only the flow run table, hence dynamic loading is the way to go. I always wanted to do dynamic loading but we could not use it since the previous version of hbase was not allowing system classes to be loaded as dynamically loaded coprocessors. So it's not a new thought, it was always the intention but was not possible due to hbase version dependency. maybe I'm bit on the paranoid side, but checking whether it is operating on the right table seems like a reasonable sanity check nonetheless; do we need to remove this check? Keeping the check won't break anything. But it will slow it down. Since this is a table level coprocessor, the check is unnecessary and at high loads, the performance would likely be affected. Do we need to move the package for this and TestFlowDataGenerator? I'm curious as to why that move is necessary. The diffs are now deleting the old files and creating the new files so it's not so easy to see what has changed. These two classes DataGeneratorForTest and TestFlowDataGenerator are common to most of the unit tests and they should have been put in a common package but for some reason, we did not do that earlier. This move feels correct to me. Regarding the diffs, I understand the changes are not easily understandable since the files moved. I think I can upload a diff of the two files to make it easier for review, let me know.
        Hide
        sjlee0 Sangjin Lee added a comment -

        Thanks for the patch Vrushali C!

        As for the unit test failures, they are related with the changes in the patch. DataGeneratorForTests assumes the presence of the timelineservice jar under its target directory, but that is true only if mvn package was done on that module. The unit tests don't require that. We need a different way to get at the location of the jar (coming from the maven local cache).

        Question: I see we're making a change to upload and use the jar that has the co-processor from hdfs, rather than adding it to the hbase install. Is that change needed for the dynamic co-processor? What is the benefit of doing that over using it out of the local install?

        (FlowRunCoprocessor.java)

        • maybe I'm bit on the paranoid side, but checking whether it is operating on the right table seems like a reasonable sanity check nonetheless; do we need to remove this check?

        (DataGeneratorForTest.java)

        • Do we need to move the package for this and TestFlowDataGenerator? I'm curious as to why that move is necessary. The diffs are now deleting the old files and creating the new files so it's not so easy to see what has changed.
        Show
        sjlee0 Sangjin Lee added a comment - Thanks for the patch Vrushali C ! As for the unit test failures, they are related with the changes in the patch. DataGeneratorForTests assumes the presence of the timelineservice jar under its target directory, but that is true only if mvn package was done on that module. The unit tests don't require that. We need a different way to get at the location of the jar (coming from the maven local cache). Question: I see we're making a change to upload and use the jar that has the co-processor from hdfs, rather than adding it to the hbase install. Is that change needed for the dynamic co-processor? What is the benefit of doing that over using it out of the local install? (FlowRunCoprocessor.java) maybe I'm bit on the paranoid side, but checking whether it is operating on the right table seems like a reasonable sanity check nonetheless; do we need to remove this check? (DataGeneratorForTest.java) Do we need to move the package for this and TestFlowDataGenerator ? I'm curious as to why that move is necessary. The diffs are now deleting the old files and creating the new files so it's not so easy to see what has changed.
        Hide
        vrushalic Vrushali C added a comment -

        Looking into the build environment to understand the mini cluster/hdfs failure

        Show
        vrushalic Vrushali C added a comment - Looking into the build environment to understand the mini cluster/hdfs failure
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
        0 mvndep 0m 54s Maven dependency ordering for branch
        +1 mvninstall 6m 44s YARN-5355 passed
        +1 compile 2m 18s YARN-5355 passed
        +1 checkstyle 0m 40s YARN-5355 passed
        +1 mvnsite 1m 20s YARN-5355 passed
        +1 mvneclipse 0m 49s YARN-5355 passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 1m 29s YARN-5355 passed
        +1 javadoc 0m 51s YARN-5355 passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 2s the patch passed
        +1 compile 2m 14s the patch passed
        +1 javac 2m 14s the patch passed
        +1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217)
        +1 mvnsite 1m 16s the patch passed
        +1 mvneclipse 0m 45s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 48s the patch passed
        +1 unit 0m 24s hadoop-yarn-api in the patch passed.
        +1 unit 0m 44s hadoop-yarn-server-timelineservice in the patch passed.
        -1 unit 2m 19s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        +1 unit 0m 9s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        35m 33s



        Reason Tests
        Failed junit tests hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
          hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
          hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
          hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
          hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
          hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
          hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-6094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847668/YARN-6094-YARN-5355.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 45cccc5f7547 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5355 / e1bdba7
        Default Java 1.8.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14664/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14664/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14664/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. 0 mvndep 0m 54s Maven dependency ordering for branch +1 mvninstall 6m 44s YARN-5355 passed +1 compile 2m 18s YARN-5355 passed +1 checkstyle 0m 40s YARN-5355 passed +1 mvnsite 1m 20s YARN-5355 passed +1 mvneclipse 0m 49s YARN-5355 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 29s YARN-5355 passed +1 javadoc 0m 51s YARN-5355 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 2m 14s the patch passed +1 javac 2m 14s the patch passed +1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 3 fixed = 214 total (was 217) +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 44s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 0m 24s hadoop-yarn-api in the patch passed. +1 unit 0m 44s hadoop-yarn-server-timelineservice in the patch passed. -1 unit 2m 19s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. +1 unit 0m 9s hadoop-yarn-site in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 35m 33s Reason Tests Failed junit tests hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity   hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-6094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847668/YARN-6094-YARN-5355.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 45cccc5f7547 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5355 / e1bdba7 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14664/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14664/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14664/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vrushalic Vrushali C added a comment -

        Uploading patch v002. Patch 001 was missing the new test directory and it's files, hence the compilation error was reported by Hadoop QA.

        To recap, the changes in the patch are:

        • documentation updates to reflect that this is now a dynamically loaded table level coprocessor (hence no restart of Region servers needed, nor any additions of coprocessor class name in hbase-site.xml)
        • in FlowRunCoprocessor, the check for isFlowRunRegion is no longer needed since this coprocessor is only enabled for one table - the flow run table.
        • Yarn config now has a new parameter that helps set the location on hdfs for the coprocessor jar
        • unit test changes.
          • Notable change is for createSchema function that has been added to DataGeneratorForTest. This function now creates the timeline service schema for the unit tests. It copies the jar to hdfs on the minicluster during the test execution and then creates the tables.
          • moved the DataGeneratorForTest and TestFlowDataGenerator classes to a common package
          • Some of the other changes in unit test file DataGeneratorForTest are for fixing the checkstyle warning of function length > 150.
        Show
        vrushalic Vrushali C added a comment - Uploading patch v002. Patch 001 was missing the new test directory and it's files, hence the compilation error was reported by Hadoop QA. To recap, the changes in the patch are: documentation updates to reflect that this is now a dynamically loaded table level coprocessor (hence no restart of Region servers needed, nor any additions of coprocessor class name in hbase-site.xml) in FlowRunCoprocessor, the check for isFlowRunRegion is no longer needed since this coprocessor is only enabled for one table - the flow run table. Yarn config now has a new parameter that helps set the location on hdfs for the coprocessor jar unit test changes. Notable change is for createSchema function that has been added to DataGeneratorForTest. This function now creates the timeline service schema for the unit tests. It copies the jar to hdfs on the minicluster during the test execution and then creates the tables. moved the DataGeneratorForTest and TestFlowDataGenerator classes to a common package Some of the other changes in unit test file DataGeneratorForTest are for fixing the checkstyle warning of function length > 150.
        Hide
        vrushalic Vrushali C added a comment -

        Looking into the jenkins failures.

        Show
        vrushalic Vrushali C added a comment - Looking into the jenkins failures.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 0m 53s Maven dependency ordering for branch
        +1 mvninstall 7m 26s YARN-5355 passed
        +1 compile 2m 31s YARN-5355 passed
        +1 checkstyle 0m 41s YARN-5355 passed
        +1 mvnsite 1m 28s YARN-5355 passed
        +1 mvneclipse 1m 2s YARN-5355 passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 1m 59s YARN-5355 passed
        +1 javadoc 1m 0s YARN-5355 passed
        0 mvndep 0m 11s Maven dependency ordering for patch
        -1 mvninstall 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        -1 compile 2m 50s hadoop-yarn in the patch failed.
        -1 javac 2m 50s hadoop-yarn in the patch failed.
        -0 checkstyle 0m 54s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 214 unchanged - 3 fixed = 215 total (was 217)
        -1 mvnsite 0m 19s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        +1 mvneclipse 0m 51s the patch passed
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 2m 10s the patch passed
        +1 javadoc 1m 1s the patch passed
        +1 unit 0m 33s hadoop-yarn-api in the patch passed.
        +1 unit 0m 54s hadoop-yarn-server-timelineservice in the patch passed.
        -1 unit 0m 19s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed.
        +1 unit 0m 9s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        38m 9s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-6094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847474/YARN-6094-YARN-5355.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4c5d0ee219b7 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5355 / e1bdba7
        Default Java 1.8.0_111
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14659/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14659/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 7m 26s YARN-5355 passed +1 compile 2m 31s YARN-5355 passed +1 checkstyle 0m 41s YARN-5355 passed +1 mvnsite 1m 28s YARN-5355 passed +1 mvneclipse 1m 2s YARN-5355 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 59s YARN-5355 passed +1 javadoc 1m 0s YARN-5355 passed 0 mvndep 0m 11s Maven dependency ordering for patch -1 mvninstall 0m 14s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. -1 compile 2m 50s hadoop-yarn in the patch failed. -1 javac 2m 50s hadoop-yarn in the patch failed. -0 checkstyle 0m 54s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 214 unchanged - 3 fixed = 215 total (was 217) -1 mvnsite 0m 19s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. +1 mvneclipse 0m 51s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 2m 10s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 0m 33s hadoop-yarn-api in the patch passed. +1 unit 0m 54s hadoop-yarn-server-timelineservice in the patch passed. -1 unit 0m 19s hadoop-yarn-server-timelineservice-hbase-tests in the patch failed. +1 unit 0m 9s hadoop-yarn-site in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 38m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-6094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847474/YARN-6094-YARN-5355.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4c5d0ee219b7 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5355 / e1bdba7 Default Java 1.8.0_111 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14659/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14659/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14659/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vrushalic Vrushali C added a comment -

        Uploading the patch file with the correct file name for branch YARN-5355

        Show
        vrushalic Vrushali C added a comment - Uploading the patch file with the correct file name for branch YARN-5355
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 4s YARN-6094 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue YARN-6094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847458/YARN-6094.001.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14657/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s YARN-6094 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-6094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847458/YARN-6094.001.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/14657/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vrushalic Vrushali C added a comment - - edited

        Uploading patch v001.

        The changes that the patch introduces are:

        • documentation updates to reflect that this is now a dynamically loaded table level coprocessor (hence no restart of Region servers needed, nor any additions of coprocessor class name in hbase-site.xml)
        • in FlowRunCoprocessor, the check for isFlowRunRegion is no longer needed since this coprocessor is only enabled for one table - the flow run table.
        • Yarn config now has a new parameter that helps set the location on hdfs for the coprocessor jar
        • unit test changes.
          • Notable change is for createSchema function that has been added to DataGeneratorForTest. This function now creates the timeline service schema for the unit tests. It copies the jar to hdfs on the minicluster during the test execution and then creates the tables.
          • moved the DataGeneratorForTest and TestFlowDataGenerator classes to a common package
          • Some of the other changes in unit test file DataGeneratorForTest are for fixing the checkstyle warning of function length > 150.
        Show
        vrushalic Vrushali C added a comment - - edited Uploading patch v001. The changes that the patch introduces are: documentation updates to reflect that this is now a dynamically loaded table level coprocessor (hence no restart of Region servers needed, nor any additions of coprocessor class name in hbase-site.xml) in FlowRunCoprocessor, the check for isFlowRunRegion is no longer needed since this coprocessor is only enabled for one table - the flow run table. Yarn config now has a new parameter that helps set the location on hdfs for the coprocessor jar unit test changes. Notable change is for createSchema function that has been added to DataGeneratorForTest. This function now creates the timeline service schema for the unit tests. It copies the jar to hdfs on the minicluster during the test execution and then creates the tables. moved the DataGeneratorForTest and TestFlowDataGenerator classes to a common package Some of the other changes in unit test file DataGeneratorForTest are for fixing the checkstyle warning of function length > 150.

          People

          • Assignee:
            vrushalic Vrushali C
            Reporter:
            vrushalic Vrushali C
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development