Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Make us build against hadoop trunk (3.0)

      Description

      Building trunk with hadoop.profile=3.0 gives exceptions (see [1]) due to change in the hadoop maven modules naming (and also usage of 3.0-SNAPSHOT instead of 3.0.0-SNAPSHOT in hbase-common).

      I can provide a patch that would move most of hadoop dependencies in their respective profiles and will define the correct hadoop deps in the 3.0 profile.

      Please tell me if that's ok to go this way.

      Thx, Eric

      [1]
      $ mvn clean install -Dhadoop.profile=3.0
      [INFO] Scanning for projects...
      [ERROR] The build could not read 3 projects -> [Help 1]
      [ERROR]
      [ERROR] The project org.apache.hbase:hbase-server:0.95-SNAPSHOT (/d/hbase.svn/hbase-server/pom.xml) has 3 errors
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 655, column 21
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 659, column 21
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 663, column 21
      [ERROR]
      [ERROR] The project org.apache.hbase:hbase-common:0.95-SNAPSHOT (/d/hbase.svn/hbase-common/pom.xml) has 3 errors
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 170, column 21
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 174, column 21
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 178, column 21
      [ERROR]
      [ERROR] The project org.apache.hbase:hbase-it:0.95-SNAPSHOT (/d/hbase.svn/hbase-it/pom.xml) has 3 errors
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 220, column 18
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 224, column 21
      [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 228, column 21
      [ERROR]

      1. 6581.txt
        11 kB
        stack
      2. 6581-009.txt
        1 kB
        Ted Yu
      3. 6581-010.txt
        4 kB
        Ted Yu
      4. 6581v2.txt
        10 kB
        stack
      5. HBASE-6581.diff
        25 kB
        Ivan Kelly
      6. HBASE-6581.diff
        18 kB
        Ivan Kelly
      7. HBASE-6581-1.patch
        9 kB
        Eric Charles
      8. HBASE-6581-2.patch
        15 kB
        Eric Charles
      9. HBASE-6581-20130821.patch
        6.44 MB
        Eric Charles
      10. HBASE-6581-3.patch
        30 kB
        Eric Charles
      11. HBASE-6581-4.patch
        31 kB
        Eric Charles
      12. HBASE-6581-5.patch
        34 kB
        Eric Charles
      13. HBASE-6581-6.patch
        40 kB
        Eric Charles
      14. HBASE-6581-7.patch
        41 kB
        Eric Charles
      15. HBASE-6581-8-pom.patch
        29 kB
        Eric Charles

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          I can provide a patch that would move most of hadoop dependencies in their respective profiles and will define the correct hadoop deps in the 3.0 profile.

          @Eric, sounds good. Put something up and we will take a look.

          Show
          Andrew Purtell added a comment - I can provide a patch that would move most of hadoop dependencies in their respective profiles and will define the correct hadoop deps in the 3.0 profile. @Eric, sounds good. Put something up and we will take a look.
          Hide
          Eric Charles added a comment -

          Existing hadoop-1.0 and hadoop-2.0 profiles untouched.

          Only changes on hadoop-3.0 profile.

          Install and test:
          mvn clean install -Dhadoop.profile=3.0 -P runSmallTests
          is Successful (tests also)

          hadoop-minicluster now defined with scope test, which is good.

          Package:
          mvn package assembly:assembly -DskipTests -Dhadoop.profile=3.0
          Tests the package against hadoop 3.0 snapshot installation with simple put/get/scan via hbase shell: Success

          In lib folder, only needed hadoop-*:
          hadoop-annotations-3.0.0-SNAPSHOT.jar
          hadoop-auth-3.0.0-SNAPSHOT.jar
          hadoop-common-3.0.0-SNAPSHOT.jar
          hadoop-hdfs-3.0.0-SNAPSHOT.jar
          hadoop-mapreduce-client-common-3.0.0-SNAPSHOT.jar
          hadoop-mapreduce-client-core-3.0.0-SNAPSHOT.jar
          hadoop-yarn-api-3.0.0-SNAPSHOT.jar
          hadoop-yarn-common-3.0.0-SNAPSHOT.jar
          hadoop-yarn-server-common-3.0.0-SNAPSHOT.jar (not sure of this one)

          Side note:
          New hbase-hadoop3-compat module which imports hbase-hadoop2-compat and excludes hadoop-* jars of version 2.
          Sounds not logical that hbase-hadoop3-compat relies on hbase-hadoop2-compat.
          In fact, hbase-hadoop1-compat and hbase-hadoop2-compat could be renamed hbase-metrics1-compat and hbase-metrics2-compat as they are pure metrics related atm.

          Show
          Eric Charles added a comment - Existing hadoop-1.0 and hadoop-2.0 profiles untouched. Only changes on hadoop-3.0 profile. Install and test: mvn clean install -Dhadoop.profile=3.0 -P runSmallTests is Successful (tests also) hadoop-minicluster now defined with scope test, which is good. Package: mvn package assembly:assembly -DskipTests -Dhadoop.profile=3.0 Tests the package against hadoop 3.0 snapshot installation with simple put/get/scan via hbase shell: Success In lib folder, only needed hadoop-*: hadoop-annotations-3.0.0-SNAPSHOT.jar hadoop-auth-3.0.0-SNAPSHOT.jar hadoop-common-3.0.0-SNAPSHOT.jar hadoop-hdfs-3.0.0-SNAPSHOT.jar hadoop-mapreduce-client-common-3.0.0-SNAPSHOT.jar hadoop-mapreduce-client-core-3.0.0-SNAPSHOT.jar hadoop-yarn-api-3.0.0-SNAPSHOT.jar hadoop-yarn-common-3.0.0-SNAPSHOT.jar hadoop-yarn-server-common-3.0.0-SNAPSHOT.jar (not sure of this one) Side note: New hbase-hadoop3-compat module which imports hbase-hadoop2-compat and excludes hadoop-* jars of version 2. Sounds not logical that hbase-hadoop3-compat relies on hbase-hadoop2-compat. In fact, hbase-hadoop1-compat and hbase-hadoop2-compat could be renamed hbase-metrics1-compat and hbase-metrics2-compat as they are pure metrics related atm.
          Hide
          Eric Charles added a comment -

          Also works with m2eclipse. Import with profile='!hadoop-1.0, hadoop-3.0'

          Show
          Eric Charles added a comment - Also works with m2eclipse. Import with profile='!hadoop-1.0, hadoop-3.0'
          Hide
          Ivan Kelly added a comment -

          The hbase-hadoop3-compat module seems to be missing from the patch.

          Show
          Ivan Kelly added a comment - The hbase-hadoop3-compat module seems to be missing from the patch.
          Hide
          Eric Charles added a comment -

          Hi Ivan,
          HBASE-6581-2.patch has now the hadoop3-compat module.
          Thx for your review,
          Eric

          Show
          Eric Charles added a comment - Hi Ivan, HBASE-6581 -2.patch has now the hadoop3-compat module. Thx for your review, Eric
          Hide
          Ivan Kelly added a comment -

          Have you done a full build with 3.0.0-SNAPSHOT?

          hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java fails to build due to
          https://github.com/apache/hadoop-common/commit/68aab040955acd483794dbf1def0b12ac5ff59e8

          Show
          Ivan Kelly added a comment - Have you done a full build with 3.0.0-SNAPSHOT? hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java fails to build due to https://github.com/apache/hadoop-common/commit/68aab040955acd483794dbf1def0b12ac5ff59e8
          Hide
          Eric Charles added a comment -

          Ivan, from what I have experienced, building hbase depending on hadoop3-SNAPSHOT is tricky in the sense that hadoop source trunk is evolving.

          I have come to the conclusion that maven build pulls the most-updated snapshot jars from the repo, but this can give "inconsistencies" and break the hbase build when you enable hadoop.profile=3.0.

          I have overcome this by installing hadoop locally with 'mvn install -DskipTests', and after packaging hbase with the maven -o option so no dep is fetched from the network and I am sure the deps it depends on are not fetched from somewhere else.

          A the time I have submitted the patch, I was sure everything was fine: compilation and even working with a hadoop3 yarn cluster.

          Now, I should retest everything, but I have just changed job and country yesterday so I prolly won't be able to do it within the coming days.

          Show
          Eric Charles added a comment - Ivan, from what I have experienced, building hbase depending on hadoop3-SNAPSHOT is tricky in the sense that hadoop source trunk is evolving. I have come to the conclusion that maven build pulls the most-updated snapshot jars from the repo, but this can give "inconsistencies" and break the hbase build when you enable hadoop.profile=3.0. I have overcome this by installing hadoop locally with 'mvn install -DskipTests', and after packaging hbase with the maven -o option so no dep is fetched from the network and I am sure the deps it depends on are not fetched from somewhere else. A the time I have submitted the patch, I was sure everything was fine: compilation and even working with a hadoop3 yarn cluster. Now, I should retest everything, but I have just changed job and country yesterday so I prolly won't be able to do it within the coming days.
          Hide
          Ivan Kelly added a comment -

          Ah ok, I'll have a go tomorrow, see what I can do.

          Show
          Ivan Kelly added a comment - Ah ok, I'll have a go tomorrow, see what I can do.
          Hide
          Eric Charles added a comment -

          Just tried it quickly, and I now have symbol : method setHostName(java.lang.String) [ERROR] location: class org.apache.hadoop.hdfs.protocol.DatanodeInfo

          Ivan, I will fix it in the coming days, if you don't do it before me

          Show
          Eric Charles added a comment - Just tried it quickly, and I now have symbol : method setHostName(java.lang.String) [ERROR] location: class org.apache.hadoop.hdfs.protocol.DatanodeInfo Ivan, I will fix it in the coming days, if you don't do it before me
          Hide
          Ivan Kelly added a comment -

          #setHostName() no longer exists. Patch dodges around this using reflection now.

          Show
          Ivan Kelly added a comment - #setHostName() no longer exists. Patch dodges around this using reflection now.
          Hide
          Ivan Kelly added a comment -

          Builds for 3.0 now. Some tests fail, though I guess it's to be expected. With default profile, it works fine.

          Show
          Ivan Kelly added a comment - Builds for 3.0 now. Some tests fail, though I guess it's to be expected. With default profile, it works fine.
          Hide
          Ted Yu added a comment -

          Meanwhile, I think we should maintain no more than 3 test failures for hadoop-2.0 profile.

          Show
          Ted Yu added a comment - Meanwhile, I think we should maintain no more than 3 test failures for hadoop-2.0 profile.
          Hide
          Ivan Kelly added a comment -

          Actually, the original patch missed src/assembly/hadoop-three-compat.xml also.

          Show
          Ivan Kelly added a comment - Actually, the original patch missed src/assembly/hadoop-three-compat.xml also.
          Hide
          Ted Yu added a comment -

          Here is the command I used:

          nohup mvn clean -Dhadoop.profile=3.0 -P localTests test > suite.txt &

          TestConstraint failed with:

          org.apache.hadoop.hbase.constraint.TestConstraint  Time elapsed: 0 sec  <<< ERROR!
          java.io.IOException: Shutting down
                  at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:229)
                  at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:92)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:688)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:661)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:656)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:598)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:567)
                  at org.apache.hadoop.hbase.constraint.TestConstraint.setUpBeforeClass(TestConstraint.java:60)
          ...
          Caused by: java.lang.RuntimeException: Master not initialized after 200 seconds
                  at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:208)
                  at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:424)
                  at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:222)
          
          Show
          Ted Yu added a comment - Here is the command I used: nohup mvn clean -Dhadoop.profile=3.0 -P localTests test > suite.txt & TestConstraint failed with: org.apache.hadoop.hbase.constraint.TestConstraint Time elapsed: 0 sec <<< ERROR! java.io.IOException: Shutting down at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:229) at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:92) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:688) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:661) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:656) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:598) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:567) at org.apache.hadoop.hbase.constraint.TestConstraint.setUpBeforeClass(TestConstraint.java:60) ... Caused by: java.lang.RuntimeException: Master not initialized after 200 seconds at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:208) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:424) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:222)
          Hide
          Ted Yu added a comment -

          160 test failures.
          Here is partial list:

            testDelayedDeleteOnFailure(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): test timed out after 25000 milliseconds
            org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion: Shutting down
            testMasterShutdown(org.apache.hadoop.hbase.master.TestMasterShutdown): Shutting down
            testMasterFailoverWithMockedRIT(org.apache.hadoop.hbase.master.TestMasterFailover): test timed out after 180000 milliseconds
            testMasterFailoverWithMockedRITOnDeadRS(org.apache.hadoop.hbase.master.TestMasterFailover): test timed out after 180000 milliseconds
            testShouldCheckMasterFailOverWhenMETAIsInOpenedState(org.apache.hadoop.hbase.master.TestMasterFailover): test timed out after 180000 milliseconds
            testSimpleMasterFailover(org.apache.hadoop.hbase.master.TestMasterFailover): Shutting down
            testRestartClusterAfterKill(org.apache.hadoop.hbase.master.TestRestartCluster): Shutting down
            testClusterRestart(org.apache.hadoop.hbase.master.TestRestartCluster): Shutting down
            org.apache.hadoop.hbase.master.TestMaster: Shutting down
            testOpenedRegionHandlerOnMasterRestart(org.apache.hadoop.hbase.master.TestOpenedRegionHandler): Shutting down
            org.apache.hadoop.hbase.master.TestMasterTransitions: Shutting down
            org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster: Shutting down
            org.apache.hadoop.hbase.rest.TestTableResource: Shutting down
            org.apache.hadoop.hbase.rest.client.TestRemoteAdmin: Shutting down
            org.apache.hadoop.hbase.rest.client.TestRemoteTable: Shutting down
            org.apache.hadoop.hbase.rest.client.TestRemoteTable
            org.apache.hadoop.hbase.rest.TestScannerResource: Shutting down
            org.apache.hadoop.hbase.rest.TestVersionResource: Shutting down
            org.apache.hadoop.hbase.rest.TestGzipFilter: Shutting down
            org.apache.hadoop.hbase.rest.TestScannersWithFilters: Shutting down
            org.apache.hadoop.hbase.rest.TestSchemaResource: Shutting down
            org.apache.hadoop.hbase.rest.TestRowResource: Shutting down
            org.apache.hadoop.hbase.rest.TestStatusResource: Shutting down
            org.apache.hadoop.hbase.rest.TestMultiRowResource: Shutting down
            testMultiClusters(org.apache.hadoop.hbase.TestHBaseTestingUtility): test timed out after 180000 milliseconds
            testMiniCluster(org.apache.hadoop.hbase.TestHBaseTestingUtility): Shutting down
            testMultipleStartStop(org.apache.hadoop.hbase.TestHBaseTestingUtility): Shutting down
            testMiniDFSCluster(org.apache.hadoop.hbase.TestHBaseTestingUtility): Port in use: localhost:0
            testSetupClusterTestBuildDir(org.apache.hadoop.hbase.TestHBaseTestingUtility): Port in use: localhost:0
          
          Tests run: 1210, Failures: 0, Errors: 160, Skipped: 2
          
          Show
          Ted Yu added a comment - 160 test failures. Here is partial list: testDelayedDeleteOnFailure(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): test timed out after 25000 milliseconds org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion: Shutting down testMasterShutdown(org.apache.hadoop.hbase.master.TestMasterShutdown): Shutting down testMasterFailoverWithMockedRIT(org.apache.hadoop.hbase.master.TestMasterFailover): test timed out after 180000 milliseconds testMasterFailoverWithMockedRITOnDeadRS(org.apache.hadoop.hbase.master.TestMasterFailover): test timed out after 180000 milliseconds testShouldCheckMasterFailOverWhenMETAIsInOpenedState(org.apache.hadoop.hbase.master.TestMasterFailover): test timed out after 180000 milliseconds testSimpleMasterFailover(org.apache.hadoop.hbase.master.TestMasterFailover): Shutting down testRestartClusterAfterKill(org.apache.hadoop.hbase.master.TestRestartCluster): Shutting down testClusterRestart(org.apache.hadoop.hbase.master.TestRestartCluster): Shutting down org.apache.hadoop.hbase.master.TestMaster: Shutting down testOpenedRegionHandlerOnMasterRestart(org.apache.hadoop.hbase.master.TestOpenedRegionHandler): Shutting down org.apache.hadoop.hbase.master.TestMasterTransitions: Shutting down org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster: Shutting down org.apache.hadoop.hbase. rest .TestTableResource: Shutting down org.apache.hadoop.hbase. rest .client.TestRemoteAdmin: Shutting down org.apache.hadoop.hbase. rest .client.TestRemoteTable: Shutting down org.apache.hadoop.hbase. rest .client.TestRemoteTable org.apache.hadoop.hbase. rest .TestScannerResource: Shutting down org.apache.hadoop.hbase. rest .TestVersionResource: Shutting down org.apache.hadoop.hbase. rest .TestGzipFilter: Shutting down org.apache.hadoop.hbase. rest .TestScannersWithFilters: Shutting down org.apache.hadoop.hbase. rest .TestSchemaResource: Shutting down org.apache.hadoop.hbase. rest .TestRowResource: Shutting down org.apache.hadoop.hbase. rest .TestStatusResource: Shutting down org.apache.hadoop.hbase. rest .TestMultiRowResource: Shutting down testMultiClusters(org.apache.hadoop.hbase.TestHBaseTestingUtility): test timed out after 180000 milliseconds testMiniCluster(org.apache.hadoop.hbase.TestHBaseTestingUtility): Shutting down testMultipleStartStop(org.apache.hadoop.hbase.TestHBaseTestingUtility): Shutting down testMiniDFSCluster(org.apache.hadoop.hbase.TestHBaseTestingUtility): Port in use: localhost:0 testSetupClusterTestBuildDir(org.apache.hadoop.hbase.TestHBaseTestingUtility): Port in use: localhost:0 Tests run: 1210, Failures: 0, Errors: 160, Skipped: 2
          Hide
          Ivan Kelly added a comment -

          New patch fixes two major issues which were causing test failures. Needs to be run with latest hadoop trunk due to HADOOP-8725

          Show
          Ivan Kelly added a comment - New patch fixes two major issues which were causing test failures. Needs to be run with latest hadoop trunk due to HADOOP-8725
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542287/HBASE-6581.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.fs.TestBlockReorder

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542287/HBASE-6581.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.fs.TestBlockReorder Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2679//console This message is automatically generated.
          Hide
          Jean-Daniel Cryans added a comment -

          Patch went a little stale, unmarking as available.

          Show
          Jean-Daniel Cryans added a comment - Patch went a little stale, unmarking as available.
          Hide
          Eric Charles added a comment -

          Attached a new patch which work with current trunk.

          Upon the poms, we also have now to deal with the protobuf 2.5.0 (used by hadoop, hbase uses 2.4.1, the communication between mixed protobufs is not possible) and also the change of FSDataOutputStream#sync method (now #hsync).

          1. To solve the protobuf issue, I propose to remove the generated classes from hbase-protocol/src/main/java and let the generation occur at build-time with hadoop-maven-plugin. This implies that the correct protoc version must be installed on the build machine (2.4.1 for pre-h3 to ensure backwards compatibility, 2.5.0 for h3).

          2. I have added a hbase-hadoop3-compat module that reuses the hbase-hadoop2-compat to avoid duplicating all the metrics classes. To allow the build, I have quickly implemented the call the the (h)sync via introspection (IMHO subject to review/optimization - another option would be to duplicate the compat classes, or to create extra modules only for that specific case).

          The patch allows me to build/assemble with the various profiles. hadoop3 profile assembly works well for me on top of an hadoop trunk build.

          There are a few test failures, but I also had some failures on my env even with a vanilla trunk (will further look at these).

          Show
          Eric Charles added a comment - Attached a new patch which work with current trunk. Upon the poms, we also have now to deal with the protobuf 2.5.0 (used by hadoop, hbase uses 2.4.1, the communication between mixed protobufs is not possible) and also the change of FSDataOutputStream#sync method (now #hsync). 1. To solve the protobuf issue, I propose to remove the generated classes from hbase-protocol/src/main/java and let the generation occur at build-time with hadoop-maven-plugin. This implies that the correct protoc version must be installed on the build machine (2.4.1 for pre-h3 to ensure backwards compatibility, 2.5.0 for h3). 2. I have added a hbase-hadoop3-compat module that reuses the hbase-hadoop2-compat to avoid duplicating all the metrics classes. To allow the build, I have quickly implemented the call the the (h)sync via introspection (IMHO subject to review/optimization - another option would be to duplicate the compat classes, or to create extra modules only for that specific case). The patch allows me to build/assemble with the various profiles. hadoop3 profile assembly works well for me on top of an hadoop trunk build. There are a few test failures, but I also had some failures on my env even with a vanilla trunk (will further look at these).
          Hide
          stack added a comment -

          The poms have 0.95-SNAPSHOT in them instead of 0.97-SNAPSHOT (or whatever current trunk is).

          Last night we updated 0.95 and trunk to pb2.5 (yeah, we should one day just gen pb classes as part of build... have you messed w/ the hadoop-maven-plugin? I didn't know such a thing existed. We should do this in another issue?)

          Otherwise, if this works for you I can commit Eric Charles (I can fix the version issue on commit np).

          Show
          stack added a comment - The poms have 0.95-SNAPSHOT in them instead of 0.97-SNAPSHOT (or whatever current trunk is). Last night we updated 0.95 and trunk to pb2.5 (yeah, we should one day just gen pb classes as part of build... have you messed w/ the hadoop-maven-plugin? I didn't know such a thing existed. We should do this in another issue?) Otherwise, if this works for you I can commit Eric Charles (I can fix the version issue on commit np).
          Hide
          Eric Charles added a comment -

          Thx Stack.

          The patch has the correct 0.97-SNAPSHOT version, isn't it the case?

          I have opened
          HBASE-9299 (Generate the protobuf classes with hadoop-maven-plugin)
          HBASE-9300 (Deal with Hadoop Syncable.sync() removal)

          depending on the result of those 2 jiras, I will upload here a new patch.

          Show
          Eric Charles added a comment - Thx Stack. The patch has the correct 0.97-SNAPSHOT version, isn't it the case? I have opened HBASE-9299 (Generate the protobuf classes with hadoop-maven-plugin) HBASE-9300 (Deal with Hadoop Syncable.sync() removal) depending on the result of those 2 jiras, I will upload here a new patch.
          Hide
          Eric Charles added a comment -

          HBASE-6581-3.patch is the correct patch (I uploaded a wrong one previously).

          This one contains the pom changes and also an attempt to cover HBASE-9300 in the new hadoop3-compat module

          Show
          Eric Charles added a comment - HBASE-6581 -3.patch is the correct patch (I uploaded a wrong one previously). This one contains the pom changes and also an attempt to cover HBASE-9300 in the new hadoop3-compat module
          Hide
          stack added a comment -

          Eric Charles Very nice (Elliott Clark – you should take a look at this patch if only to see an example of your compat modules being properly used). Let me run it by hadoopqa. Only question is did you mean to name the method synch? Or did you mean to name it hsync (or since it is now done properly, just sync?).

          Show
          stack added a comment - Eric Charles Very nice ( Elliott Clark – you should take a look at this patch if only to see an example of your compat modules being properly used). Let me run it by hadoopqa. Only question is did you mean to name the method synch? Or did you mean to name it hsync (or since it is now done properly, just sync?).
          Hide
          Eric Charles added a comment -

          Just realized that synch was not the best name... prolly sync would be better. Let's see what Elliott and hadoopqa tell about the patch before I resubmit another one.

          Show
          Eric Charles added a comment - Just realized that synch was not the best name... prolly sync would be better. Let's see what Elliott and hadoopqa tell about the patch before I resubmit another one.
          Hide
          stack added a comment -

          Smile. Yeah, poor old sync has been well abused already (sync, flush, hsync w/ variants, etc.). I provoked hadoopqa. Lets see what it says. I can commit then if you change the name... 'sync' seems to be the right name for this method (smile)

          Show
          stack added a comment - Smile. Yeah, poor old sync has been well abused already (sync, flush, hsync w/ variants, etc.). I provoked hadoopqa. Lets see what it says. I can commit then if you change the name... 'sync' seems to be the right name for this method (smile)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12599627/HBASE-6581-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12599627/HBASE-6581-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6859//console This message is automatically generated.
          Hide
          stack added a comment -

          Fix method name and I'll commit Eric Charles. Thanks

          Show
          stack added a comment - Fix method name and I'll commit Eric Charles . Thanks
          Hide
          Eric Charles added a comment -

          Now I remember the intermediate state in which I was when I coded that synch method, thinking, oh, ProtobufLogWriter has already a sync method (line 131), if I give the same name to the new one, java compiler will not be happy, and even if it is happy, dev will be confused.

          this method is supposed to sync the fsdataaoutputstream, so what about naming it syncOutputStream. syncOutputStream will blame us for its name for the rest of its life, but who cares...

          Show
          Eric Charles added a comment - Now I remember the intermediate state in which I was when I coded that synch method, thinking, oh, ProtobufLogWriter has already a sync method (line 131), if I give the same name to the new one, java compiler will not be happy, and even if it is happy, dev will be confused. this method is supposed to sync the fsdataaoutputstream, so what about naming it syncOutputStream. syncOutputStream will blame us for its name for the rest of its life, but who cares...
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12599627/HBASE-6581-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings).

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12599627/HBASE-6581-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings). +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6872//console This message is automatically generated.
          Hide
          stack added a comment -

          Would be good if you could come closer in to its implemented name (this is good on history http://blog.cloudera.com/blog/2009/07/file-appends-in-hdfs/). hflush is take? hsync is taken?

          Show
          stack added a comment - Would be good if you could come closer in to its implemented name (this is good on history http://blog.cloudera.com/blog/2009/07/file-appends-in-hdfs/ ). hflush is take? hsync is taken?
          Hide
          Eric Charles added a comment -

          new patch using hsync name + javadoc

          Show
          Eric Charles added a comment - new patch using hsync name + javadoc
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12600565/HBASE-6581-4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600565/HBASE-6581-4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6963//console This message is automatically generated.
          Hide
          stack added a comment -

          Eric Charles Did you intend the new name to be hsynch (with an 'h' suffix)? Does it have to be this to not clash w/ hsync in h3? And can we cache the method rather than do relection to find it every time? Thanks. Patch looks good otherwise.

          Show
          stack added a comment - Eric Charles Did you intend the new name to be hsynch (with an 'h' suffix)? Does it have to be this to not clash w/ hsync in h3? And can we cache the method rather than do relection to find it every time? Thanks. Patch looks good otherwise.
          Hide
          Eric Charles added a comment -

          Stack seems like I always add some h on sync. I have renamed it to hsync (no clash

          About the method cache, I mentioned earlier "subject to review/optimization". Let's do this.
          We can store the "Method method" as a field and instantiate/cache it when the ProtobufLogWriter object is constructed. Now we must ask about the thread-safety:
          1. Is the ProtobufLogWriter instance used in a multithread context?
          2. If yes, is a Method field thread safe?

          Depending on this, we should put (or not) some synchronization/locking.

          Show
          Eric Charles added a comment - Stack seems like I always add some h on sync. I have renamed it to hsync (no clash About the method cache, I mentioned earlier "subject to review/optimization". Let's do this. We can store the "Method method" as a field and instantiate/cache it when the ProtobufLogWriter object is constructed. Now we must ask about the thread-safety: 1. Is the ProtobufLogWriter instance used in a multithread context? 2. If yes, is a Method field thread safe? Depending on this, we should put (or not) some synchronization/locking.
          Hide
          stack added a comment -

          Sorry, missed that Eric. Yes, PBLW is used in multithreaded context. Ain't sure about thread-safety – I'd presume so since we do this caching elsewhere and that the 'invocation' is going to do what happens anyways on a method invocation. Regards sync, ain't there some in the layers above this method? Thanks.

          Show
          stack added a comment - Sorry, missed that Eric. Yes, PBLW is used in multithreaded context. Ain't sure about thread-safety – I'd presume so since we do this caching elsewhere and that the 'invocation' is going to do what happens anyways on a method invocation. Regards sync, ain't there some in the layers above this method? Thanks.
          Hide
          Eric Charles added a comment -
          • FSHLog#syncer has some synchronized block, but still one call to PBLW.sync() is not in a synchronized block. so this maybe a problem. I guess we can't change like that the unsynchronized code block FSHLog (line 1068) without side-effect (the logic seems there quite tricky).
          • I looked around (web, book...) to know if method.invoke(...) is safe. It sounds to me it is not like a direct method call, the state of that object may change and may thus be unsafe (no idea). Maybe someone can shed light on this?
          • I looked in the hbase code for other Method objects as field. I found one in TableRecordReaderImpl where is getCounter method is stored as field. No particular guard is there when invoke(...) is called on it. But that's for sure a less sensible function and maybe the threadsafety is naturally ensured by the calle (nextKeyValue).
          • Side note: this method object came because I wanted to avoid duplicating all the hadoop2-compat classes in the hadoop3-compat module. If we agree to copy/paste them, the method object and our questions disappear (although I would like to have answers on them). I don't anyway duplicating is the way to go.
          Show
          Eric Charles added a comment - FSHLog#syncer has some synchronized block, but still one call to PBLW.sync() is not in a synchronized block. so this maybe a problem. I guess we can't change like that the unsynchronized code block FSHLog (line 1068) without side-effect (the logic seems there quite tricky). I looked around (web, book...) to know if method.invoke(...) is safe. It sounds to me it is not like a direct method call, the state of that object may change and may thus be unsafe (no idea). Maybe someone can shed light on this? I looked in the hbase code for other Method objects as field. I found one in TableRecordReaderImpl where is getCounter method is stored as field. No particular guard is there when invoke(...) is called on it. But that's for sure a less sensible function and maybe the threadsafety is naturally ensured by the calle (nextKeyValue). Side note: this method object came because I wanted to avoid duplicating all the hadoop2-compat classes in the hadoop3-compat module. If we agree to copy/paste them, the method object and our questions disappear (although I would like to have answers on them). I don't anyway duplicating is the way to go.
          Hide
          Eric Charles added a comment -

          We already have a FSDataOutputStream as field. This one is not synchronized and not thread-safe by nature (I guess). So caller are thus supposed to do the safety job.

          Stack Waiting on your green light to submit a new patch without further synchronization on the method object call.

          Show
          Eric Charles added a comment - We already have a FSDataOutputStream as field. This one is not synchronized and not thread-safe by nature (I guess). So caller are thus supposed to do the safety job. Stack Waiting on your green light to submit a new patch without further synchronization on the method object call.
          Hide
          stack added a comment -

          I'd imagine we'd get the method once on startup and then it would never change during the life of the WAL?

          FSHLog#syncer has some synchronized block, but still one call to PBLW.sync() is not in a synchronized block. so this maybe a problem. I guess we can't change like that the unsynchronized code block FSHLog (line 1068) without side-effect (the logic seems there quite tricky).

          I'm not the expert here. The less synchronization hereabouts the better but current state comes of study by others.

          Copy/paste is probably fine especially if you feel it 'safer'.

          Show
          stack added a comment - I'd imagine we'd get the method once on startup and then it would never change during the life of the WAL? FSHLog#syncer has some synchronized block, but still one call to PBLW.sync() is not in a synchronized block. so this maybe a problem. I guess we can't change like that the unsynchronized code block FSHLog (line 1068) without side-effect (the logic seems there quite tricky). I'm not the expert here. The less synchronization hereabouts the better but current state comes of study by others. Copy/paste is probably fine especially if you feel it 'safer'.
          Hide
          Eric Charles added a comment -

          new patch with hsync name + lazy instanciate of the method + cache method

          Ho additional synchronization, as existing FSDataOutputStream field is not synchronized, relying on callers synchronization.

          Show
          Eric Charles added a comment - new patch with hsync name + lazy instanciate of the method + cache method Ho additional synchronization, as existing FSDataOutputStream field is not synchronized, relying on callers synchronization.
          Hide
          Eric Charles added a comment -

          Stack don't commit for now, I have an issue running hserver with that last patch (not sure if I messed my env with all those various hadoop/hbase versions, or the 2 weeks changes I just pulled made things break here...).

          Anyway, any feedback on the last patch always welcome.

          Show
          Eric Charles added a comment - Stack don't commit for now, I have an issue running hserver with that last patch (not sure if I messed my env with all those various hadoop/hbase versions, or the 2 weeks changes I just pulled made things break here...). Anyway, any feedback on the last patch always welcome.
          Hide
          Eric Charles added a comment -

          Stack Sorry for the noise, finally the hanging connections, tablealreadyexistsexception, method not found... issues I had yesterday were manifestations of badly configured envs. It works fine today (I had to define the HBASE_CLASSPATH having the hbase-hadoop3-compat jar so it is found first on the classpath).

          Before committing, someone should test the patch on hadoop2/3 (works here, but better double-check).

          Show
          Eric Charles added a comment - Stack Sorry for the noise, finally the hanging connections, tablealreadyexistsexception, method not found... issues I had yesterday were manifestations of badly configured envs. It works fine today (I had to define the HBASE_CLASSPATH having the hbase-hadoop3-compat jar so it is found first on the classpath). Before committing, someone should test the patch on hadoop2/3 (works here, but better double-check).
          Hide
          stack added a comment -

          Making critical 0.98. Could come into 0.96 too. Just needs someone taking it for a run on cluster making sure it basically works.

          Show
          stack added a comment - Making critical 0.98. Could come into 0.96 too. Just needs someone taking it for a run on cluster making sure it basically works.
          Hide
          Eric Charles added a comment -

          Any further progress on this one? Patch risks to be obsolete with the time. Thx.

          Show
          Eric Charles added a comment - Any further progress on this one? Patch risks to be obsolete with the time. Thx.
          Hide
          stack added a comment -

          The patch looks good.

          I probably asked this before, but is this right for hadoop1?

          + public void hsync() throws IOException {
          + output.sync();

          The sync does the right thing? I thought it was hsync of hflush or something in h1?

          Similar in h2. Is the sync function called 'sync'?

          I supppose I should check myself rather than ask you Eric.

          ... and even looking at your patch you look to be doing the right thing.

          Let me get back to this. Its rotted. I should fix it up since we let it rot.

          Show
          stack added a comment - The patch looks good. I probably asked this before, but is this right for hadoop1? + public void hsync() throws IOException { + output.sync(); The sync does the right thing? I thought it was hsync of hflush or something in h1? Similar in h2. Is the sync function called 'sync'? I supppose I should check myself rather than ask you Eric. ... and even looking at your patch you look to be doing the right thing. Let me get back to this. Its rotted. I should fix it up since we let it rot.
          Hide
          Andrew Purtell added a comment -

          Hadoop 3.0 isn't released, downgrading this to normal ("Major") priority. Also moving out of 0.98.0 as there is no patch.

          Show
          Andrew Purtell added a comment - Hadoop 3.0 isn't released, downgrading this to normal ("Major") priority. Also moving out of 0.98.0 as there is no patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12601057/HBASE-6581-5.patch
          against trunk revision .
          ATTACHMENT ID: 12601057

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8221//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12601057/HBASE-6581-5.patch against trunk revision . ATTACHMENT ID: 12601057 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8221//console This message is automatically generated.
          Hide
          stack added a comment -

          Any change in an issues status now seems to trigger an hadoopqa run. Am I imagining this? Anyone know what changed?

          Show
          stack added a comment - Any change in an issues status now seems to trigger an hadoopqa run. Am I imagining this? Anyone know what changed?
          Hide
          Eric Charles added a comment -

          HBASE-6581-6.patch rebased to latest trunk.

          Show
          Eric Charles added a comment - HBASE-6581 -6.patch rebased to latest trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12622372/HBASE-6581-6.patch
          against trunk revision .
          ATTACHMENT ID: 12622372

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8384//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12622372/HBASE-6581-6.patch against trunk revision . ATTACHMENT ID: 12622372 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8384//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Looks like patch v6 was generated incorrecly: hadoop-three-compat.xml should be a new file.

          --- a/hbase-assembly/src/main/assembly/hadoop-three-compat.xml
          +++ /dev/null
          @@ -1,46 +0,0 @@
          -<?xml version="1.0"?>
          
          Show
          Ted Yu added a comment - Looks like patch v6 was generated incorrecly: hadoop-three-compat.xml should be a new file. --- a/hbase-assembly/src/main/assembly/hadoop-three-compat.xml +++ /dev/ null @@ -1,46 +0,0 @@ -<?xml version= "1.0" ?>
          Hide
          Eric Charles added a comment -

          Ted Yu Thx for the review and sorry, I messed my git command.

          Attached another v7.
          I did successfully
          svn co https://svn.apache.org/repos/asf/hbase/trunk
          patch -p0 < HBASE-6581-7.patch
          mvn clean install -DskipTests -Dhadoop.profile=3.0$

          The generated hbase dist works on a hadoop (trunk) cluster.

          Show
          Eric Charles added a comment - Ted Yu Thx for the review and sorry, I messed my git command. Attached another v7. I did successfully svn co https://svn.apache.org/repos/asf/hbase/trunk patch -p0 < HBASE-6581 -7.patch mvn clean install -DskipTests -Dhadoop.profile=3.0$ The generated hbase dist works on a hadoop (trunk) cluster.
          Hide
          Eric Charles added a comment -

          Ted Yu oops, master does not start today. I will check and fix. you can already review the pach and see if it compiles with -Dhadoop.profile=3.0

          Show
          Eric Charles added a comment - Ted Yu oops, master does not start today. I will check and fix. you can already review the pach and see if it compiles with -Dhadoop.profile=3.0
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12622528/HBASE-6581-7.patch
          against trunk revision .
          ATTACHMENT ID: 12622528

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          +<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1 http://maven.apache.org/xsd/assembly-1.1.1.xsd">
          + <!--This 'all' id is not appended to the produced bundle because we do this: http://maven.apache.org/plugins/maven-assembly-plugin/faq.html#required-classifiers -->
          + <!-This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.->
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <!-This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.->
          + <!-This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.->
          + <!-This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.->

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12622528/HBASE-6581-7.patch against trunk revision . ATTACHMENT ID: 12622528 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: +<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1 http://maven.apache.org/xsd/assembly-1.1.1.xsd "> + <!--This 'all' id is not appended to the produced bundle because we do this: http://maven.apache.org/plugins/maven-assembly-plugin/faq.html#required-classifiers --> + <!- This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. -> +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <!- This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. -> + <!- This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. -> + <!- This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. -> -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8389//console This message is automatically generated.
          Hide
          Ted Yu added a comment -
          Tests run: 1760, Failures: 0, Errors: 0, Skipped: 23
          

          Compared with https://builds.apache.org/job/PreCommit-HBASE-Build/8388/console :

          Tests run: 1768, Failures: 0, Errors: 0, Skipped: 23
          

          Looks like 8 tests were missing in PreCommit build #8389

          Show
          Ted Yu added a comment - Tests run: 1760, Failures: 0, Errors: 0, Skipped: 23 Compared with https://builds.apache.org/job/PreCommit-HBASE-Build/8388/console : Tests run: 1768, Failures: 0, Errors: 0, Skipped: 23 Looks like 8 tests were missing in PreCommit build #8389
          Hide
          Eric Charles added a comment -

          Strange, the patch does not disable any test.
          Btw, the today HMaster startup failure on h3 comes from a null value when jetty web server is started. I will open on jira on hadoop-common and will report back here.

          Show
          Eric Charles added a comment - Strange, the patch does not disable any test. Btw, the today HMaster startup failure on h3 comes from a null value when jetty web server is started. I will open on jira on hadoop-common and will report back here.
          Hide
          Ted Yu added a comment -

          If you perform a diff between build 8388 and 8389 on hbase-server module, you will find out which test(s) hung.

          Show
          Ted Yu added a comment - If you perform a diff between build 8388 and 8389 on hbase-server module, you will find out which test(s) hung.
          Hide
          Eric Charles added a comment -

          I have taken the logs but a diff via the command line does not help (timestamps...) - I have to do a manual review of all tests. Will do it tomorrow.

          Show
          Eric Charles added a comment - I have taken the logs but a diff via the command line does not help (timestamps...) - I have to do a manual review of all tests. Will do it tomorrow.
          Hide
          Ted Yu added a comment -

          Here is what I did:

          • extract the tests for hbase-server module for both runs.
            e.g. here is beginning:
            Running org.apache.hadoop.hbase.replication.TestReplicationStateZKImpl
            Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.968 sec
            Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager
            
          • Sort the tests:
            grep 'org.apache.hadoop.' qa-8388.txt | sort > qa-8388.sorted
            
          • perform diff between the sorted lists:
            $ diff qa-8388.sorted qa-8389.sorted
            211d210
            < Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationHLogReaderManager
            

            You can run TestReplicationHLogReaderManager locally to see what happens.

          Show
          Ted Yu added a comment - Here is what I did: extract the tests for hbase-server module for both runs. e.g. here is beginning: Running org.apache.hadoop.hbase.replication.TestReplicationStateZKImpl Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.968 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager Sort the tests: grep 'org.apache.hadoop.' qa-8388.txt | sort > qa-8388.sorted perform diff between the sorted lists: $ diff qa-8388.sorted qa-8389.sorted 211d210 < Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationHLogReaderManager You can run TestReplicationHLogReaderManager locally to see what happens.
          Hide
          Eric Charles added a comment -

          Thx Ted for the tip about Maven output logs processing.

          On my laptop:

          A/
          git checkout trunk
          mvn clean test -Dtest=TestReplicationHLogReaderManager
          SUCCESS 3:19.245s

          B/
          git checkout HBASE-6581-hadoop3
          mvn clean test -Dtest=TestReplicationHLogReaderManager
          ... hangs for a long time. Not patient enough, CTRL^C...

          C/
          and with profile 3.0
          git checkout HBASE-6581-hadoop3
          mvn clean test -Dtest=TestReplicationHLogReaderManager -Dhadoop.profile=3.0
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (default-test) on project hbase-server: Failure or timeout -> [Help 1]

          more hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.regionserver.TestReplicationHLogReaderManager.txt

          The
          2014-01-12 19:59:09,942 INFO [IPC Server handler 6 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43180 is added to blk_1073741828_1004

          {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[[DISK]DS-f41c8aca-a032-4865-8fa9-2367efc6ae03:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-2b59d0b7-f913-45c7-9927-12a6d619c054:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-fd05c8d9-a047-4a77-bce9-b1899c1dc 823:NORMAL|RBW]]}

          size 7
          2014-01-12 19:59:09,945 INFO [IPC Server handler 7 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45050 is added to blk_1073741828_1004

          {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[[DISK]DS-f41c8aca-a032-4865-8fa9-2367efc6ae03:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-2b59d0b7-f913-45c7-9927-12a6d619c054:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-fd05c8d9-a047-4a77-bce9-b1899c1dc 823:NORMAL|RBW]]}

          size 7
          2014-01-12 19:59:09,945 INFO [IPC Server handler 8 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48792 is added to blk_1073741828_1004

          {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[[DISK]DS-f41c8aca-a032-4865-8fa9-2367efc6ae03:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-2b59d0b7-f913-45c7-9927-12a6d619c054:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-fd05c8d9-a047-4a77-bce9-b1899c1dc 823:NORMAL|RBW]]}

          size 7
          2014-01-12 19:59:09,976 DEBUG [main] wal.FSHLog(883): Moved 3 WAL file(s) to /user/eric/hbase/oldWALs
          2014-01-12 19:59:10,016 INFO [main] hbase.ResourceChecker(171): after: replication.regionserver.TestReplicationHLogReaderManager#test[0] Thread=113 (was 111)
          Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@25242963
          java.lang.Thread.sleep(Native Method)
          org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:252)
          org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:39)
          org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:135)
          java.lang.Thread.run(Thread.java:724)

          Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-590648032_1 at /127.0.0.1:56548 Waiting for operation #3
          sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
          sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
          sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
          org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
          org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
          org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
          java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
          java.io.BufferedInputStream.read(BufferedInputStream.java:254)
          java.io.DataInputStream.readShort(DataInputStream.java:312)
          org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:52)
          org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:198)
          java.lang.Thread.run(Thread.java:724)

          • Thread LEAK? -, OpenFileDescriptor=233 (was 212) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=4096 (was 4096), SystemLoadAverage=307 (was 290) - SystemLoadAverage LEAK? -, ProcessCount=263 (was 263), AvailableMemoryMB=2163 (was 2222), Co
            nnectionCount=0 (was 0)
            2014-01-12 19:59:10,036 INFO [main] hbase.ResourceChecker(147): before: replication.regionserver.TestReplicationHLogReaderManager#test[1] Thread=113, OpenFileDescriptor=233, MaxFileDescriptor=4096, SystemLoadAverage=307, ProcessCount=263, Ava
            ilableMemoryMB=2163, ConnectionCount=0
            2014-01-12 19:59:10,036 INFO [main] wal.FSHLog(396): WAL/HLog configuration: blocksize=128 MB, rollsize=121.60 MB, enabled=true
            2014-01-12 19:59:10,078 INFO [main] wal.FSHLog(575): New WAL /user/eric/hbase/test/some+server.1389553150040
            2014-01-12 19:59:10,260 INFO [IPC Server handler 1 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43180 is added to blk_1073741829_1005 {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[[DISK]DS-3c6d935d-9ed2-4d21-be0b-60165402737b:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-9bd0dd0d-e3ba-4fcc-8ba5-c37b1463c262:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-0c960596-2901-4baf-8a50-a66be764b 376:NORMAL|RBW]]}

            size 7
            2014-01-12 19:59:10,262 INFO [IPC Server handler 0 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48792 is added to blk_1073741829_1005{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re
            plicas=[ReplicaUnderConstruction[[DISK]DS-3c6d935d-9ed2-4d21-be0b-60165402737b:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-9bd0dd0d-e3ba-4fcc-8ba5-c37b1463c262:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-0c960596-2901-4baf-8a50-a66be764b

          Show
          Eric Charles added a comment - Thx Ted for the tip about Maven output logs processing. On my laptop: A/ git checkout trunk mvn clean test -Dtest=TestReplicationHLogReaderManager SUCCESS 3:19.245s B/ git checkout HBASE-6581 -hadoop3 mvn clean test -Dtest=TestReplicationHLogReaderManager ... hangs for a long time. Not patient enough, CTRL^C... C/ and with profile 3.0 git checkout HBASE-6581 -hadoop3 mvn clean test -Dtest=TestReplicationHLogReaderManager -Dhadoop.profile=3.0 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK- HBASE-2 :test (default-test) on project hbase-server: Failure or timeout -> [Help 1] more hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.regionserver.TestReplicationHLogReaderManager.txt The 2014-01-12 19:59:09,942 INFO [IPC Server handler 6 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43180 is added to blk_1073741828_1004 {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[[DISK]DS-f41c8aca-a032-4865-8fa9-2367efc6ae03:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-2b59d0b7-f913-45c7-9927-12a6d619c054:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-fd05c8d9-a047-4a77-bce9-b1899c1dc 823:NORMAL|RBW]]} size 7 2014-01-12 19:59:09,945 INFO [IPC Server handler 7 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45050 is added to blk_1073741828_1004 {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[[DISK]DS-f41c8aca-a032-4865-8fa9-2367efc6ae03:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-2b59d0b7-f913-45c7-9927-12a6d619c054:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-fd05c8d9-a047-4a77-bce9-b1899c1dc 823:NORMAL|RBW]]} size 7 2014-01-12 19:59:09,945 INFO [IPC Server handler 8 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48792 is added to blk_1073741828_1004 {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[[DISK]DS-f41c8aca-a032-4865-8fa9-2367efc6ae03:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-2b59d0b7-f913-45c7-9927-12a6d619c054:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-fd05c8d9-a047-4a77-bce9-b1899c1dc 823:NORMAL|RBW]]} size 7 2014-01-12 19:59:09,976 DEBUG [main] wal.FSHLog(883): Moved 3 WAL file(s) to /user/eric/hbase/oldWALs 2014-01-12 19:59:10,016 INFO [main] hbase.ResourceChecker(171): after: replication.regionserver.TestReplicationHLogReaderManager#test [0] Thread=113 (was 111) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@25242963 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:252) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:39) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:135) java.lang.Thread.run(Thread.java:724) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-590648032_1 at /127.0.0.1:56548 Waiting for operation #3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:52) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:198) java.lang.Thread.run(Thread.java:724) Thread LEAK? -, OpenFileDescriptor=233 (was 212) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=4096 (was 4096), SystemLoadAverage=307 (was 290) - SystemLoadAverage LEAK? -, ProcessCount=263 (was 263), AvailableMemoryMB=2163 (was 2222), Co nnectionCount=0 (was 0) 2014-01-12 19:59:10,036 INFO [main] hbase.ResourceChecker(147): before: replication.regionserver.TestReplicationHLogReaderManager#test [1] Thread=113, OpenFileDescriptor=233, MaxFileDescriptor=4096, SystemLoadAverage=307, ProcessCount=263, Ava ilableMemoryMB=2163, ConnectionCount=0 2014-01-12 19:59:10,036 INFO [main] wal.FSHLog(396): WAL/HLog configuration: blocksize=128 MB, rollsize=121.60 MB, enabled=true 2014-01-12 19:59:10,078 INFO [main] wal.FSHLog(575): New WAL /user/eric/hbase/test/some+server.1389553150040 2014-01-12 19:59:10,260 INFO [IPC Server handler 1 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43180 is added to blk_1073741829_1005 {blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[[DISK]DS-3c6d935d-9ed2-4d21-be0b-60165402737b:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-9bd0dd0d-e3ba-4fcc-8ba5-c37b1463c262:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-0c960596-2901-4baf-8a50-a66be764b 376:NORMAL|RBW]]} size 7 2014-01-12 19:59:10,262 INFO [IPC Server handler 0 on 38555] blockmanagement.BlockManager(2293): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48792 is added to blk_1073741829_1005{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, re plicas=[ReplicaUnderConstruction[ [DISK] DS-3c6d935d-9ed2-4d21-be0b-60165402737b:NORMAL|RBW], ReplicaUnderConstruction[ [DISK] DS-9bd0dd0d-e3ba-4fcc-8ba5-c37b1463c262:NORMAL|RBW], ReplicaUnderConstruction[ [DISK] DS-0c960596-2901-4baf-8a50-a66be764b
          Hide
          Ted Yu added a comment -
          +public interface Sync {
          

          Please add annotation for stability and audience

          +    public void hsync() throws IOException {
          +        try {
          +            method().invoke(output);
          +        }
          +        catch (Exception e) {
          +            throw new RuntimeException("Exception while invoking sync", e);
          +        }
          +    }
          

          Where is the field output assigned ?

          Show
          Ted Yu added a comment - + public interface Sync { Please add annotation for stability and audience + public void hsync() throws IOException { + try { + method().invoke(output); + } + catch (Exception e) { + throw new RuntimeException( "Exception while invoking sync" , e); + } + } Where is the field output assigned ?
          Hide
          Honghua Feng added a comment -
          +//  public boolean remove(K key, V value) {
          +//    return removeValue(key, value);
          +//  }
          

          why not just remove but comment out instead?

          +    /** The method name to call for Hdoop 3.x */

          Hdoop => Hadoop?

          Show
          Honghua Feng added a comment - + // public boolean remove(K key, V value) { + // return removeValue(key, value); + // } why not just remove but comment out instead? + /** The method name to call for Hdoop 3.x */ Hdoop => Hadoop?
          Hide
          Eric Charles added a comment -

          Ted Yu the field is assigned by the inheriting classes, which make the code not nice. I will update the patch with an abstract method getOutput() to make it more readable., adding the stability and audience on the interface.

          Honghua Feng Eclipse was complaining with that deprecated method (tried with jdk6, jdk7 and jdk8) the final patch should not have that comment. Although, I will open a specific jira to remove that deprecated method (maybe already opened, will check) - comment updated.

          I am still with 2 issues:
          1. TestReplicationHLogReaderManager
          2. A stupid NPE due to a null hostname when jetty webserver is kicked (due I think to a recent change in hadoop).

          Thx to all for your feedbacks & Stay tuned.

          Show
          Eric Charles added a comment - Ted Yu the field is assigned by the inheriting classes, which make the code not nice. I will update the patch with an abstract method getOutput() to make it more readable., adding the stability and audience on the interface. Honghua Feng Eclipse was complaining with that deprecated method (tried with jdk6, jdk7 and jdk8) the final patch should not have that comment. Although, I will open a specific jira to remove that deprecated method (maybe already opened, will check) - comment updated. I am still with 2 issues: 1. TestReplicationHLogReaderManager 2. A stupid NPE due to a null hostname when jetty webserver is kicked (due I think to a recent change in hadoop). Thx to all for your feedbacks & Stay tuned.
          Hide
          Eric Charles added a comment -

          The second issue mentioned above (npe) is fixed with HDFS-5760.

          Show
          Eric Charles added a comment - The second issue mentioned above (npe) is fixed with HDFS-5760 .
          Hide
          Eric Charles added a comment -

          and now I have looked at first issue related to TestReplicationHLogReaderManager. The test is working but is just slow and is probably killed by the build system. Some methods take 4 times more with the usage of the Method object. Strangely, TestHLog takes the same time - I will write a small blog post with more details.

          Bottom line: I would like to propose the commit of the changes related to the pom.xml (they were not easy to setup and I would prefer not loosing that work : it basicallyl introduces a hadoop3 module and nicely exclude the hadoop-core 1.1 and so on...). For the java class, we can further think to a another solution.

          If someone gives the green light for the poms, I will upload a new patch.

          Show
          Eric Charles added a comment - and now I have looked at first issue related to TestReplicationHLogReaderManager. The test is working but is just slow and is probably killed by the build system. Some methods take 4 times more with the usage of the Method object. Strangely, TestHLog takes the same time - I will write a small blog post with more details. Bottom line: I would like to propose the commit of the changes related to the pom.xml (they were not easy to setup and I would prefer not loosing that work : it basicallyl introduces a hadoop3 module and nicely exclude the hadoop-core 1.1 and so on...). For the java class, we can further think to a another solution. If someone gives the green light for the poms, I will upload a new patch.
          Hide
          Andrew Purtell added a comment -

          Does this issue confuse issues with Hadoop 3.0 and JDK 8?

          Some methods take 4 times more with the usage of the Method object. Strangely, TestHLog takes the same time - I will write a small blog post with more details.

          Consider elaborating a bit here.

          Show
          Andrew Purtell added a comment - Does this issue confuse issues with Hadoop 3.0 and JDK 8? Some methods take 4 times more with the usage of the Method object. Strangely, TestHLog takes the same time - I will write a small blog post with more details. Consider elaborating a bit here.
          Hide
          Ted Yu added a comment -

          propose the commit of the changes related to the pom.xml

          For the java class, we can further think to a another solution.

          I think we should do above.

          Show
          Ted Yu added a comment - propose the commit of the changes related to the pom.xml For the java class, we can further think to a another solution. I think we should do above.
          Hide
          Eric Charles added a comment -

          Ted Yu Further to previous comment, attached HBASE-6581-8-pom.patch which contains only the changes related to the poms to support hadoop-3-snapshot.

          I also have another patch with the java code, but I first need to figure out why TestReplicationHLogReaderManager takes more time.

          If jenkins is happy with HBASE-6581-8-pom.patch, it can be safely committed (no impact on TestReplicationHLogReaderManager or anything else)

          Show
          Eric Charles added a comment - Ted Yu Further to previous comment, attached HBASE-6581 -8-pom.patch which contains only the changes related to the poms to support hadoop-3-snapshot. I also have another patch with the java code, but I first need to figure out why TestReplicationHLogReaderManager takes more time. If jenkins is happy with HBASE-6581 -8-pom.patch, it can be safely committed (no impact on TestReplicationHLogReaderManager or anything else)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12624766/HBASE-6581-8-pom.patch
          against trunk revision .
          ATTACHMENT ID: 12624766

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          +<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1 http://maven.apache.org/xsd/assembly-1.1.1.xsd">
          + <!--This 'all' id is not appended to the produced bundle because we do this: http://maven.apache.org/plugins/maven-assembly-plugin/faq.html#required-classifiers -->
          + <!-This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.->
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <!-This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.->
          + <!-This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.->
          + <!-This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.->

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12624766/HBASE-6581-8-pom.patch against trunk revision . ATTACHMENT ID: 12624766 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: +<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1 http://maven.apache.org/xsd/assembly-1.1.1.xsd "> + <!--This 'all' id is not appended to the produced bundle because we do this: http://maven.apache.org/plugins/maven-assembly-plugin/faq.html#required-classifiers --> + <!- This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. -> +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <!- This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. -> + <!- This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. -> + <!- This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. -> -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8508//console This message is automatically generated.
          Hide
          Eric Charles added a comment -

          didn't 'mvn site' - will check and submit new patch if needed - otherwise, any other feedback on the pom changes?

          Show
          Eric Charles added a comment - didn't 'mvn site' - will check and submit new patch if needed - otherwise, any other feedback on the pom changes?
          Hide
          Eric Charles added a comment -

          my 'mvn site' simply hangs (as well on with or without this patch)

          [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
          [INFO] Forking HBase - Shell 0.99.0-SNAPSHOT
          [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
          [INFO] 
          [INFO] --- build-helper-maven-plugin:1.5:add-source (jspcSource-packageInfo-source) @ hbase-shell ---
          [INFO] Source directory: /home/eric/wrk/com/hbase.git/hbase-shell/target/generated-jamon added.
          [INFO] Source directory: /home/eric/wrk/com/hbase.git/hbase-shell/target/generated-sources/java added.
          [INFO] 
          [INFO] <<< maven-javadoc-plugin:2.9.1:aggregate (report:aggregate) @ hbase <<<
          [INFO] Parent project loaded from repository: org.apache:apache:pom:12
          [INFO] Relativizing decoration links with respect to project URL: http://hbase.apache.org
          [INFO] Rendering site with org.apache.maven.skins:maven-stylus-skin:jar:1.5 skin.
          [INFO] Skipped "About" report, file "index.html" already exists for the English version.
          [INFO] Skipped "Source Xref" report, file "xref/index.html" already exists for the English version.
          [INFO] Skipped "Test Source Xref" report, file "xref-test/index.html" already exists for the English version.
          [WARNING] <title> was already defined in <properties>, ignored <title> in <head>.
          [WARNING] <title> was already defined in <properties>, ignored <title> in <head>.
          [INFO] Generating "Project Team" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Mailing Lists" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Continuous Integration" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Issue Tracking" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Project License" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Source Repository" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Project Summary" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Dependency Information" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Project Modules" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Dependency Management" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Dependencies" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Dependency Convergence" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Plugin Management" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Project Plugins" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Distribution Management" report    --- maven-project-info-reports-plugin:2.7
          [INFO] Generating "Source Xref" report    --- maven-jxr-plugin:2.3
          [INFO] Generating "Test Source Xref" report    --- maven-jxr-plugin:2.3
          [INFO] Generating "JavaDocs" report    --- maven-javadoc-plugin:2.9.1
          
          >>> Hangs...
          
          Show
          Eric Charles added a comment - my 'mvn site' simply hangs (as well on with or without this patch) [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking HBase - Shell 0.99.0-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- build-helper-maven-plugin:1.5:add-source (jspcSource-packageInfo-source) @ hbase-shell --- [INFO] Source directory: /home/eric/wrk/com/hbase.git/hbase-shell/target/generated-jamon added. [INFO] Source directory: /home/eric/wrk/com/hbase.git/hbase-shell/target/generated-sources/java added. [INFO] [INFO] <<< maven-javadoc-plugin:2.9.1:aggregate (report:aggregate) @ hbase <<< [INFO] Parent project loaded from repository: org.apache:apache:pom:12 [INFO] Relativizing decoration links with respect to project URL: http: //hbase.apache.org [INFO] Rendering site with org.apache.maven.skins:maven-stylus-skin:jar:1.5 skin. [INFO] Skipped "About" report, file "index.html" already exists for the English version. [INFO] Skipped "Source Xref" report, file "xref/index.html" already exists for the English version. [INFO] Skipped "Test Source Xref" report, file "xref-test/index.html" already exists for the English version. [WARNING] <title> was already defined in <properties>, ignored <title> in <head>. [WARNING] <title> was already defined in <properties>, ignored <title> in <head>. [INFO] Generating "Project Team" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Mailing Lists" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Continuous Integration" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Issue Tracking" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Project License" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Source Repository" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Project Summary" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Dependency Information" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Project Modules" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Dependency Management" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Dependencies" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Dependency Convergence" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Plugin Management" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Project Plugins" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Distribution Management" report --- maven-project-info-reports-plugin:2.7 [INFO] Generating "Source Xref" report --- maven-jxr-plugin:2.3 [INFO] Generating "Test Source Xref" report --- maven-jxr-plugin:2.3 [INFO] Generating "JavaDocs" report --- maven-javadoc-plugin:2.9.1 >>> Hangs...
          Hide
          stack added a comment -

          Don't mind the failing mvn site. It is not you. The hanging you see probably completes eventually or it will OOME.

          Show
          stack added a comment - Don't mind the failing mvn site. It is not you. The hanging you see probably completes eventually or it will OOME.
          Hide
          stack added a comment -

          Does HBASE-10336 have to go in first Eric Charles before this one?

          Show
          stack added a comment - Does HBASE-10336 have to go in first Eric Charles before this one?
          Hide
          Eric Charles added a comment -

          Stack no, HBASE-10336 is orthogonal and has nothing to do with this one, so you can commit this when you like.
          I don't know why 'mvn site' is reported as failed by jenkins (couldn't replay it cause it hangs on my env with vanilla trunk also)

          Show
          Eric Charles added a comment - Stack no, HBASE-10336 is orthogonal and has nothing to do with this one, so you can commit this when you like. I don't know why 'mvn site' is reported as failed by jenkins (couldn't replay it cause it hangs on my env with vanilla trunk also)
          Hide
          Eric Charles added a comment -

          stack oh, just saw your previous comment on OOME. Is this a normal behavior, has this already been discussed? should I open a JIRA for this?

          Show
          Eric Charles added a comment - stack oh, just saw your previous comment on OOME. Is this a normal behavior, has this already been discussed? should I open a JIRA for this?
          Hide
          Eric Charles added a comment -

          stack With HBASE-6581-8-pom.patch., 'mvn site -Dmaven.javadoc.skip=true' produces valid website.

          Show
          Eric Charles added a comment - stack With HBASE-6581 -8-pom.patch., 'mvn site -Dmaven.javadoc.skip=true' produces valid website.
          Hide
          Eric Charles added a comment -

          Now that hadoop-1 support is removed from trunk, this jira will be a bit easier to solve, at least with the number of test to perform while developing it.oo
          I will take over this again next week to double check the difference between hadoop 2.4 and 3.0-SNAPSHOT.
          The shim support is still something we should achieve to avoid issues later on and HBASE-10336 (Remove deprecated usage of Hadoop HttpServer in InfoServer) will be needed before going forward here.

          Show
          Eric Charles added a comment - Now that hadoop-1 support is removed from trunk, this jira will be a bit easier to solve, at least with the number of test to perform while developing it.oo I will take over this again next week to double check the difference between hadoop 2.4 and 3.0-SNAPSHOT. The shim support is still something we should achieve to avoid issues later on and HBASE-10336 (Remove deprecated usage of Hadoop HttpServer in InfoServer) will be needed before going forward here.
          Hide
          stack added a comment -

          Thanks Eric Charles

          Show
          stack added a comment - Thanks Eric Charles
          Hide
          Andrew Purtell added a comment -

          Maybe we only need work on this in trunk? Is there a need for this with 0.98 or earlier?

          Show
          Andrew Purtell added a comment - Maybe we only need work on this in trunk? Is there a need for this with 0.98 or earlier?
          Hide
          Eric Charles added a comment -

          Andrew Purtell I was just focusing on trunk. To my view, the first step would be to make it work in trunk. I can rebase and reupload a patch which would integrate also HBASE-10336 (10336 is a precondition to this). To avoid further exploding the patch list, just let me know if someone can test it (at least let Jenkins test it).

          Show
          Eric Charles added a comment - Andrew Purtell I was just focusing on trunk. To my view, the first step would be to make it work in trunk. I can rebase and reupload a patch which would integrate also HBASE-10336 (10336 is a precondition to this). To avoid further exploding the patch list, just let me know if someone can test it (at least let Jenkins test it).
          Hide
          stack added a comment -

          Eric Charles I can add a hadoop3 profile to apache jenkins, np.

          Show
          stack added a comment - Eric Charles I can add a hadoop3 profile to apache jenkins, np.
          Hide
          stack added a comment -

          Lets get this fixed for hbase-1.0. YARN-2032 wants it. See also HADOOP-10995. Attached patch gets us past HttpServer issue and changes sync calls to hsync (sync was deprecated in hadoop 0.21). Cheats by reusing hadoop2-compat rather than add a hadoop3-compat. Could fix this later?

          I now get the below:

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 10.168s
          [INFO] Finished at: Fri Sep 12 15:49:22 PDT 2014
          [INFO] Final Memory: 39M/452M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hbase-common: Compilation failure: Compilation failure:
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[32,31] error: package org.apache.hadoop.mapred does not exist
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[33,34] error: package org.apache.hadoop.mapreduce does not exist
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[104,65] error: cannot find symbol
          [ERROR] symbol:   class Job
          [ERROR] location: class User
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[113,45] error: cannot find symbol
          [ERROR] symbol:   class JobConf
          [ERROR] location: class User
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[272,58] error: cannot find symbol
          [ERROR] symbol:   class Job
          [ERROR] location: class SecureHadoopUser
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[297,38] error: cannot find symbol
          [ERROR] symbol:   class JobConf
          [ERROR] location: class SecureHadoopUser
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java:[46,61] Unsafe is internal proprietary API and may be removed in a future release
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:[1157,19] Unsafe is internal proprietary API and may be removed in a future release
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:[1163,21] Unsafe is internal proprietary API and may be removed in a future release
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:[1168,28] Unsafe is internal proprietary API and may be removed in a future release
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[279,16] error: cannot find symbol
          [ERROR] symbol:   class Job
          [ERROR] location: class SecureHadoopUser
          [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[303,24] error: cannot find symbol
          [ERROR] -> [Help 1]
          [ERROR]
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR]
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR]
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hbase-common
          kalashnikov:hbase.git stack$ git diff > 6581.txt
          

          Will be back to this.

          Show
          stack added a comment - Lets get this fixed for hbase-1.0. YARN-2032 wants it. See also HADOOP-10995 . Attached patch gets us past HttpServer issue and changes sync calls to hsync (sync was deprecated in hadoop 0.21). Cheats by reusing hadoop2-compat rather than add a hadoop3-compat. Could fix this later? I now get the below: [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10.168s [INFO] Finished at: Fri Sep 12 15:49:22 PDT 2014 [INFO] Final Memory: 39M/452M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile ( default -compile) on project hbase-common: Compilation failure: Compilation failure: [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[32,31] error: package org.apache.hadoop.mapred does not exist [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[33,34] error: package org.apache.hadoop.mapreduce does not exist [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[104,65] error: cannot find symbol [ERROR] symbol: class Job [ERROR] location: class User [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[113,45] error: cannot find symbol [ERROR] symbol: class JobConf [ERROR] location: class User [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[272,58] error: cannot find symbol [ERROR] symbol: class Job [ERROR] location: class SecureHadoopUser [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[297,38] error: cannot find symbol [ERROR] symbol: class JobConf [ERROR] location: class SecureHadoopUser [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java:[46,61] Unsafe is internal proprietary API and may be removed in a future release [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:[1157,19] Unsafe is internal proprietary API and may be removed in a future release [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:[1163,21] Unsafe is internal proprietary API and may be removed in a future release [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:[1168,28] Unsafe is internal proprietary API and may be removed in a future release [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[279,16] error: cannot find symbol [ERROR] symbol: class Job [ERROR] location: class SecureHadoopUser [ERROR] /Users/stack/checkouts/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java:[303,24] error: cannot find symbol [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch . [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http: //cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hbase-common kalashnikov:hbase.git stack$ git diff > 6581.txt Will be back to this.
          Hide
          stack added a comment -

          Anyone have input on how to do the above migrations? Appreciated. Else will start digging.

          Show
          stack added a comment - Anyone have input on how to do the above migrations? Appreciated. Else will start digging.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12668498/6581.txt
          against trunk revision .
          ATTACHMENT ID: 12668498

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestClassFinder

          -1 core zombie tests. There are 2 zombie test(s): at org.apache.oozie.test.MiniHCatServer$1.run(MiniHCatServer.java:137)
          at org.apache.oozie.test.XTestCase$MiniClusterShutdownMonitor.run(XTestCase.java:1071)
          at org.apache.oozie.test.XTestCase.waitFor(XTestCase.java:692)
          at org.apache.oozie.test.XTestCase.sleep(XTestCase.java:710)
          at org.apache.oozie.service.TestCoordMaterializeTriggerService.testCoordMaterializeTriggerService2(TestCoordMaterializeTriggerService.java:102)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668498/6581.txt against trunk revision . ATTACHMENT ID: 12668498 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.TestClassFinder -1 core zombie tests . There are 2 zombie test(s): at org.apache.oozie.test.MiniHCatServer$1.run(MiniHCatServer.java:137) at org.apache.oozie.test.XTestCase$MiniClusterShutdownMonitor.run(XTestCase.java:1071) at org.apache.oozie.test.XTestCase.waitFor(XTestCase.java:692) at org.apache.oozie.test.XTestCase.sleep(XTestCase.java:710) at org.apache.oozie.service.TestCoordMaterializeTriggerService.testCoordMaterializeTriggerService2(TestCoordMaterializeTriggerService.java:102) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10867//console This message is automatically generated.
          Hide
          stack added a comment -

          Meld in some of Eric's patch. A lot has changed since his patch (its easier now). Avoiding a h3 compat if I can.

          Show
          stack added a comment - Meld in some of Eric's patch. A lot has changed since his patch (its easier now). Avoiding a h3 compat if I can.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12668508/6581v2.txt
          against trunk revision .
          ATTACHMENT ID: 12668508

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint

          -1 core zombie tests. There are 5 zombie test(s): at org.apache.phoenix.end2end.DistinctCountIT.testDistinctCountOnIndexTab(DistinctCountIT.java:471)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668508/6581v2.txt against trunk revision . ATTACHMENT ID: 12668508 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint -1 core zombie tests . There are 5 zombie test(s): at org.apache.phoenix.end2end.DistinctCountIT.testDistinctCountOnIndexTab(DistinctCountIT.java:471) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10868//console This message is automatically generated.
          Hide
          Eric Charles added a comment -

          stack do you like me to digg into this or you have all in hand? I followed the api changes and yes, could/should be easier now.
          Just tell me, I have cycles begin next week for this.

          Show
          Eric Charles added a comment - stack do you like me to digg into this or you have all in hand? I followed the api changes and yes, could/should be easier now. Just tell me, I have cycles begin next week for this.
          Hide
          stack added a comment -

          Eric Charles That'd be excellent Eric. Looks like some fellas need this to work (YARN-2032). We should get it in before 1.0 if we can.

          The httpserver change looks like small oversight. Moving on to hsync instead of sync seems ok since deprecated since hadoop 0.21 and 0.99 is hadoop2 only (sorry you did all that work w/ Interfaces figuring which to use). I am reluctant introducing a hbase-hadoop3-compat. We will have that crazy build story again where we had script to generate the h2 and h1 artifacts (if we have to have it, h2 could be just default and you have to do extra work to make the h3).

          I got stuck on the stuff pasted above, how to do User going forward with SecureHadoopUser and hadoop User. There is no mapred package anymore and the mapreduce is in different jar.

          I'd say something that basically works would do for now. Can refine later. Thanks Eric.

          Show
          stack added a comment - Eric Charles That'd be excellent Eric. Looks like some fellas need this to work ( YARN-2032 ). We should get it in before 1.0 if we can. The httpserver change looks like small oversight. Moving on to hsync instead of sync seems ok since deprecated since hadoop 0.21 and 0.99 is hadoop2 only (sorry you did all that work w/ Interfaces figuring which to use). I am reluctant introducing a hbase-hadoop3-compat. We will have that crazy build story again where we had script to generate the h2 and h1 artifacts (if we have to have it, h2 could be just default and you have to do extra work to make the h3). I got stuck on the stuff pasted above, how to do User going forward with SecureHadoopUser and hadoop User. There is no mapred package anymore and the mapreduce is in different jar. I'd say something that basically works would do for now. Can refine later. Thanks Eric.
          Hide
          Ted Yu added a comment -

          With the attached patch, I was able to build and run the following command:

          mvn clean test -Dhadoop.profile=3.0 -PrunAllTests
          

          There was one test failure:

            TestRegionObserverInterface.testRecovery:633->verifyMethodResult:749 Result of org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver.getCtPreWALRestore is expected to be 1, while we get 0
          

          But the above failure sometimes appears in normal test run.
          The fix in TestConfServlet.java is due to fixed spelling mistake in hadoop 3.0

          Show
          Ted Yu added a comment - With the attached patch, I was able to build and run the following command: mvn clean test -Dhadoop.profile=3.0 -PrunAllTests There was one test failure: TestRegionObserverInterface.testRecovery:633->verifyMethodResult:749 Result of org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver.getCtPreWALRestore is expected to be 1, while we get 0 But the above failure sometimes appears in normal test run. The fix in TestConfServlet.java is due to fixed spelling mistake in hadoop 3.0
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12689739/6581-009.txt
          against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53.
          ATTACHMENT ID: 12689739

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/checkstyle-aggregate.html

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12689739/6581-009.txt against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53. ATTACHMENT ID: 12689739 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12279//console This message is automatically generated.
          Hide
          Eric Charles added a comment -

          Ted Yu Which patch should I try? HBASE-6581-7.patch and/or HBASE-6581-8-pom.patch ... ?

          Show
          Eric Charles added a comment - Ted Yu Which patch should I try? HBASE-6581 -7.patch and/or HBASE-6581 -8-pom.patch ... ?
          Hide
          Ted Yu added a comment -

          Please try 6581-009.txt

          Show
          Ted Yu added a comment - Please try 6581-009.txt
          Hide
          Eric Charles added a comment -

          Tried to run head based on 2.5.1 on hadoop-3.0.0-SNAPSHOT - It works fine, at least for in a pseudo-distributed environement (didn't try mapreduce).

          Building with 6581-009.txt + -DskipTests + -Dhadoop.profile=3.0 gives at the very end of the build:
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.4:single (default-cli) on project hbase-assembly: Error reading assemblies: Error locating assembly descriptor: src/main/assembly/hadoop-three-compat.xml

          It should be easy to add /hadoop-three-compat.xml

          Show
          Eric Charles added a comment - Tried to run head based on 2.5.1 on hadoop-3.0.0-SNAPSHOT - It works fine, at least for in a pseudo-distributed environement (didn't try mapreduce). Building with 6581-009.txt + -DskipTests + -Dhadoop.profile=3.0 gives at the very end of the build: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.4:single (default-cli) on project hbase-assembly: Error reading assemblies: Error locating assembly descriptor: src/main/assembly/hadoop-three-compat.xml It should be easy to add /hadoop-three-compat.xml
          Hide
          Ted Yu added a comment -

          Thanks for trying out the patch.

          Here is one with hadoop-three-compat.xml added.

          Show
          Ted Yu added a comment - Thanks for trying out the patch. Here is one with hadoop-three-compat.xml added.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12692980/6581-010.txt
          against master branch at commit 03e17168c3feab765fec26693318f4b8ae6a9468.
          ATTACHMENT ID: 12692980

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          +<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1 http://maven.apache.org/xsd/assembly-1.1.1.xsd">
          + <!--This 'all' id is not appended to the produced bundle because we do this: http://maven.apache.org/plugins/maven-assembly-plugin/faq.html#required-classifiers -->

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/checkstyle-aggregate.html

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692980/6581-010.txt against master branch at commit 03e17168c3feab765fec26693318f4b8ae6a9468. ATTACHMENT ID: 12692980 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: +<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.1 http://maven.apache.org/xsd/assembly-1.1.1.xsd "> + <!--This 'all' id is not appended to the produced bundle because we do this: http://maven.apache.org/plugins/maven-assembly-plugin/faq.html#required-classifiers --> +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12507//console This message is automatically generated.
          Hide
          Eric Charles added a comment -

          +1 for 6581-010.txt

          Compiles fines (didn't yet run the tests..., but jenkins was happy with it +1 core tests. The patch passed unit tests )
          Deployed in pseudo-distributed on hadoop-3.0.0-SNAPSHOT and works fine with basic operations I made (put/get/scan).

          One could argue that having hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar is strange (...hadoop2... should be ...hadoop3...), but let's not be too strict on this.

          Show
          Eric Charles added a comment - +1 for 6581-010.txt Compiles fines (didn't yet run the tests..., but jenkins was happy with it +1 core tests. The patch passed unit tests ) Deployed in pseudo-distributed on hadoop-3.0.0-SNAPSHOT and works fine with basic operations I made (put/get/scan). One could argue that having hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar is strange (...hadoop2... should be ...hadoop3...), but let's not be too strict on this.
          Hide
          Eric Charles added a comment -

          +1 for 6581-010.txt

          Buils fine with hadoop profile 3.0 (didn't yet run the tests..., but jenkins was happy with it +1 core tests. The patch passed unit tests )
          Deployed in pseudo-distributed on hadoop-3.0.0-SNAPSHOT and works fine with basic operations I made (put/get/scan).

          One could argue that having hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar is strange (...hadoop2... should be ...hadoop3...), but let's not be too strict on this.

          Show
          Eric Charles added a comment - +1 for 6581-010.txt Buils fine with hadoop profile 3.0 (didn't yet run the tests..., but jenkins was happy with it +1 core tests. The patch passed unit tests ) Deployed in pseudo-distributed on hadoop-3.0.0-SNAPSHOT and works fine with basic operations I made (put/get/scan). One could argue that having hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar is strange (...hadoop2... should be ...hadoop3...), but let's not be too strict on this.
          Hide
          Ted Yu added a comment -

          stack:
          What do you think of patch v10 ?

          Show
          Ted Yu added a comment - stack : What do you think of patch v10 ?

            People

            • Assignee:
              Eric Charles
              Reporter:
              Eric Charles
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development