HBase
  1. HBase
  2. HBASE-9047

Tool to handle finishing replication when the cluster is offline

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.96.0
    • Fix Version/s: 0.98.0, 0.94.15, 0.96.2, 0.99.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We're having a discussion on the mailing list about replicating the data on a cluster that was shut down in an offline fashion. The motivation could be that you don't want to bring HBase back up but still need that data on the slave.

      So I have this idea of a tool that would be running on the master cluster while it is down, although it could also run at any time. Basically it would be able to read the replication state of each master region server, finish replicating what's missing to all the slave, and then clear that state in zookeeper.

      The code that handles replication does most of that already, see ReplicationSourceManager and ReplicationSource. Basically when ReplicationSourceManager.init() is called, it will check all the queues in ZK and try to grab those that aren't attached to a region server. If the whole cluster is down, it will grab all of them.

      The beautiful thing here is that you could start that tool on all your machines and the load will be spread out, but that might not be a big concern if replication wasn't lagging since it would take a few seconds to finish replicating the missing data for each region server.

      I'm guessing when starting ReplicationSourceManager you'd give it a fake region server ID, and you'd tell it not to start its own source.

      FWIW the main difference in how replication is handled between Apache's HBase and Facebook's is that the latter is always done separately of HBase itself. This jira isn't about doing that.

      1. HBASE-9047-0.94-v1.patch
        21 kB
        Demai Ni
      2. HBASE-9047-trunk-v7.patch
        21 kB
        Demai Ni
      3. HBASE-9047-trunk-v7.patch
        21 kB
        Demai Ni
      4. HBASE-9047-trunk-v6.patch
        21 kB
        Demai Ni
      5. HBASE-9047-trunk-v5.patch
        21 kB
        Demai Ni
      6. HBASE-9047-trunk-v4.patch
        21 kB
        stack
      7. HBASE-9047-trunk-v4.patch
        21 kB
        Demai Ni
      8. HBASE-9047-trunk-v3.patch
        21 kB
        Demai Ni
      9. HBASE-9047-trunk-v2.patch
        21 kB
        Demai Ni
      10. HBASE-9047-trunk-v1.patch
        22 kB
        Demai Ni
      11. HBASE-9047-0.94.9-v0.PATCH
        23 kB
        Demai Ni
      12. HBASE-9047-trunk-v0.patch
        23 kB
        Demai Ni

        Issue Links

          Activity

          Hide
          Demai Ni added a comment -

          Lars, I am out of town now and donot have access to my laptop and cluster. I will look at it as soon as I am back after Christmas . Sorry about that

          Demai on the run

          Show
          Demai Ni added a comment - Lars, I am out of town now and donot have access to my laptop and cluster. I will look at it as soon as I am back after Christmas . Sorry about that Demai on the run
          Hide
          Lars Hofhansl added a comment -

          TestReplicationSyncUpTool still fails occasionally. Here's a run from 0.94.

          Error Message
          
          @Peer1 t1_syncup should be sync up and have 50 rows expected:<50> but was:<100>
          
          Stacktrace
          
          java.lang.AssertionError: @Peer1 t1_syncup should be sync up and have 50 rows expected:<50> but was:<100>
          	at org.junit.Assert.fail(Assert.java:93)
          	at org.junit.Assert.failNotEquals(Assert.java:647)
          	at org.junit.Assert.assertEquals(Assert.java:128)
          	at org.junit.Assert.assertEquals(Assert.java:472)
          	at org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool.mimicSyncUpAfterDelete(TestReplicationSyncUpTool.java:300)
          	at org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool.testSyncUpTool(TestReplicationSyncUpTool.java:136)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
          

          Demai Ni, could you have a quick look?

          Show
          Lars Hofhansl added a comment - TestReplicationSyncUpTool still fails occasionally. Here's a run from 0.94. Error Message @Peer1 t1_syncup should be sync up and have 50 rows expected:<50> but was:<100> Stacktrace java.lang.AssertionError: @Peer1 t1_syncup should be sync up and have 50 rows expected:<50> but was:<100> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool.mimicSyncUpAfterDelete(TestReplicationSyncUpTool.java:300) at org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool.testSyncUpTool(TestReplicationSyncUpTool.java:136) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62) Demai Ni , could you have a quick look?
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #9 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/9/)
          HBASE-9047 - addendum, include new files (larsh: rev 1551464)

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #9 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/9/ ) HBASE-9047 - addendum, include new files (larsh: rev 1551464) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in hbase-0.96-hadoop2 #152 (See https://builds.apache.org/job/hbase-0.96-hadoop2/152/)
          HBASE-9047 - addendum, include new files (larsh: rev 1551463)

          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
            HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551462)
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          Show
          Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #152 (See https://builds.apache.org/job/hbase-0.96-hadoop2/152/ ) HBASE-9047 - addendum, include new files (larsh: rev 1551463) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551462) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #14 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/14/)
          HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551465)

          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #14 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/14/ ) HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551465) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in hbase-0.96 #229 (See https://builds.apache.org/job/hbase-0.96/229/)
          HBASE-9047 - addendum, include new files (larsh: rev 1551463)

          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
            HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551462)
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          Show
          Hudson added a comment - FAILURE: Integrated in hbase-0.96 #229 (See https://builds.apache.org/job/hbase-0.96/229/ ) HBASE-9047 - addendum, include new files (larsh: rev 1551463) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551462) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98 #17 (See https://builds.apache.org/job/HBase-0.98/17/)
          HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551465)

          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #17 (See https://builds.apache.org/job/HBase-0.98/17/ ) HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551465) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94 #1229 (See https://builds.apache.org/job/HBase-0.94/1229/)
          HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551461)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1229 (See https://builds.apache.org/job/HBase-0.94/1229/ ) HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551461) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94-security #362 (See https://builds.apache.org/job/HBase-0.94-security/362/)
          HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551461)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #362 (See https://builds.apache.org/job/HBase-0.94-security/362/ ) HBASE-9047 Tool to handle finishing replication when the cluster is offline (Demai Ni) (larsh: rev 1551461) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #4730 (See https://builds.apache.org/job/HBase-TRUNK/4730/)
          HBASE-9047 - addendum, include new files (larsh: rev 1551464)

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #4730 (See https://builds.apache.org/job/HBase-TRUNK/4730/ ) HBASE-9047 - addendum, include new files (larsh: rev 1551464) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
          Hide
          Lars Hofhansl added a comment -

          Committed to all branches.
          (In trunk I botched the commit message, though)

          Show
          Lars Hofhansl added a comment - Committed to all branches. (In trunk I botched the commit message, though)
          Hide
          Lars Hofhansl added a comment -

          I'll do it today.

          Show
          Lars Hofhansl added a comment - I'll do it today.
          Hide
          stack added a comment -

          Lars Hofhansl You going to commit or would you like me too?

          Show
          stack added a comment - Lars Hofhansl You going to commit or would you like me too?
          Hide
          Lars Hofhansl added a comment -

          No automated way, currently. Need to run it yourself
          I'll run some selected tests.

          If there are no further objections, I'll commit this to all branches soon.

          Show
          Lars Hofhansl added a comment - No automated way, currently. Need to run it yourself I'll run some selected tests. If there are no further objections, I'll commit this to all branches soon.
          Hide
          Demai Ni added a comment -

          Lars Hofhansl, is there a way to run the 0.94. patch with Hadoop QA? I used the 'attach Files' feature to upload the patch, and hadoop QA assumes it is for trunk. Thanks... Demai

          Show
          Demai Ni added a comment - Lars Hofhansl , is there a way to run the 0.94. patch with Hadoop QA? I used the 'attach Files' feature to upload the patch, and hadoop QA assumes it is for trunk. Thanks... Demai
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12618538/HBASE-9047-0.94-v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8153//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618538/HBASE-9047-0.94-v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8153//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          attached patch for 0.94.

          for 0.96, the trunk patch should be able to apply. do we need create a separated 0.96 patch?

          thanks... Demai

          Show
          Demai Ni added a comment - attached patch for 0.94. for 0.96, the trunk patch should be able to apply. do we need create a separated 0.96 patch? thanks... Demai
          Hide
          Demai Ni added a comment -

          Lars Hofhansl,stack, thanks. I will upload a 0.94 patch tonight.

          Show
          Demai Ni added a comment - Lars Hofhansl , stack , thanks. I will upload a 0.94 patch tonight.
          Hide
          stack added a comment -

          +1 (Operability)

          Show
          stack added a comment - +1 (Operability)
          Hide
          Lars Hofhansl added a comment -

          +1

          Could you update the 0.94 patch?
          stack, I assume you want this in 0.96? It's mostly additional code, and quite useful in practice.

          Show
          Lars Hofhansl added a comment - +1 Could you update the 0.94 patch? stack , I assume you want this in 0.96? It's mostly additional code, and quite useful in practice.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12618460/HBASE-9047-trunk-v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618460/HBASE-9047-trunk-v7.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8145//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          since we have a good HadoopQA build now, I attached the same patch to get a clean HadoopQA run.

          Show
          Demai Ni added a comment - since we have a good HadoopQA build now, I attached the same patch to get a clean HadoopQA run.
          Hide
          Demai Ni added a comment -

          Lars Hofhansl,

          I went through the findbug issues, and didn't identify any introduced by this patch. The only one related with replications is

           
          	org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.processEndOfFile() might ignore java.io.IOException
          

          this is an existing one

          Show
          Demai Ni added a comment - Lars Hofhansl , I went through the findbug issues, and didn't identify any introduced by this patch. The only one related with replications is org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.processEndOfFile() might ignore java.io.IOException this is an existing one
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12618077/HBASE-9047-trunk-v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618077/HBASE-9047-trunk-v7.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8123//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          sorry. there was a typo of the class name. It was my bad.
          this time it works without importing or full classname on my local cluster.

          Show
          Demai Ni added a comment - sorry. there was a typo of the class name. It was my bad. this time it works without importing or full classname on my local cluster.
          Hide
          Lars Hofhansl added a comment -

          It seems you either have to import org.apache.hadoop.hbase.replication.regionserver.ReplicationSyncUp or provide a full classname.

          It also means that you have not testing this.

          Looks good to go to me when that is fixed.

          Show
          Lars Hofhansl added a comment - It seems you either have to import org.apache.hadoop.hbase.replication.regionserver.ReplicationSyncUp or provide a full classname. It also means that you have not testing this. Looks good to go to me when that is fixed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12618067/HBASE-9047-trunk-v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          -1 hadoop1.0. The patch failed to compile against the hadoop 1.0 profile.
          Here is snippet of errors:

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hbase-server: Compilation failure
          [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[503,50] package ReplicationSyncup does not exist
          [ERROR] -> [Help 1]
          org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hbase-server: Compilation failure
          /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[503,50] package ReplicationSyncup does not exist
          
          	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
          	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
          	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
          --
          Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure
          /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[503,50] package ReplicationSyncup does not exist
          
          	at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729)
          	at org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:128)
          	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8122//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618067/HBASE-9047-trunk-v7.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. -1 hadoop1.0 . The patch failed to compile against the hadoop 1.0 profile. Here is snippet of errors: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile ( default -compile) on project hbase-server: Compilation failure [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[503,50] package ReplicationSyncup does not exist [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile ( default -compile) on project hbase-server: Compilation failure /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[503,50] package ReplicationSyncup does not exist at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) -- Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[503,50] package ReplicationSyncup does not exist at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729) at org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:128) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8122//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          Lars Hofhansl, a new patch per Lars' suggestion. It is much cleaner using instanceof. thanks... Demai

          Show
          Demai Ni added a comment - Lars Hofhansl , a new patch per Lars' suggestion. It is much cleaner using instanceof. thanks... Demai
          Hide
          Lars Hofhansl added a comment - - edited

          That should work. How about if (stopper instanceof ReplicationSyncUp.Server)? Would that be better?

          Show
          Lars Hofhansl added a comment - - edited That should work. How about if (stopper instanceof ReplicationSyncUp.Server) ? Would that be better?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12617970/HBASE-9047-trunk-v6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617970/HBASE-9047-trunk-v6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8115//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          Lars Hofhansl, I added a checking against the stopper before the loop so that the code in replicatonsource will only be executed if its from replicationsyncup

          if (stopper.toString().contains("ReplicationSyncUp")) {
          .....
          }
          
          Show
          Demai Ni added a comment - Lars Hofhansl , I added a checking against the stopper before the loop so that the code in replicatonsource will only be executed if its from replicationsyncup if (stopper.toString().contains( "ReplicationSyncUp" )) { ..... }
          Hide
          Demai Ni added a comment -

          Lars Hofhansl, many thanks for the review, and good point about the performance concern. As pointed out by you, it is on the error path so only impact a small portion of the code flow. But it still will be good if I can find a way to run loop only for the syncup tool. One idea(well, not the best interface design) is to see whether insider replicationsource, the code can recognize the host name made up by the syncup tool, and use it as indicator.

          ReplicationSyncUp:DummyServer
          ...
           hostname = System.currentTimeMillis() + ".SyncUpTool.replication.org";
          ...
          

          I will give it a try. And, any suggestions are gratefully appreciated.

          Demai

          Show
          Demai Ni added a comment - Lars Hofhansl , many thanks for the review, and good point about the performance concern. As pointed out by you, it is on the error path so only impact a small portion of the code flow. But it still will be good if I can find a way to run loop only for the syncup tool. One idea(well, not the best interface design) is to see whether insider replicationsource, the code can recognize the host name made up by the syncup tool, and use it as indicator. ReplicationSyncUp:DummyServer ... hostname = System .currentTimeMillis() + ".SyncUpTool.replication.org" ; ... I will give it a try. And, any suggestions are gratefully appreciated. Demai
          Hide
          Lars Hofhansl added a comment -

          Looks good (this is actually very clever I think). The only concern is the potential cost of this:

          +          // In the case of disaster/recovery, HMaster may be shutdown/crashed before flush data
          +          // from .logs to .oldlogs. Loop into .logs folders and check whether a match exists
          +          FileStatus[] rss = fs.listStatus(manager.getLogDir());
          +          for (FileStatus rs : rss) {
          +            Path p = rs.getPath();
          +            FileStatus[] logs = fs.listStatus(p);
          ...
          

          Any way we can restrict this to the case when we run this tool? Or maybe it's not a problem since we only get here when did not find a log to begin with...?

          Show
          Lars Hofhansl added a comment - Looks good (this is actually very clever I think). The only concern is the potential cost of this: + // In the case of disaster/recovery, HMaster may be shutdown/crashed before flush data + // from .logs to .oldlogs. Loop into .logs folders and check whether a match exists + FileStatus[] rss = fs.listStatus(manager.getLogDir()); + for (FileStatus rs : rss) { + Path p = rs.getPath(); + FileStatus[] logs = fs.listStatus(p); ... Any way we can restrict this to the case when we run this tool? Or maybe it's not a problem since we only get here when did not find a log to begin with...?
          Hide
          Demai Ni added a comment -

          I went through the two -1 and didn't find the relationship with this patch

          for -1 javadoc. The javadoc tool appears to have generated 1 warning messages. Besides the existing (and known) Bytes.java about sun.misc.Unsafe, there is another one

          src/main/java/org/apache/hadoop/hbase/mapreduce/LabelExpander.java:188: warning - @return tag has no arguments
          

          for -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. I also didn't find anything associated with this patch. But there is one related with replicationsource.java

           org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.processEndOfFile() might ignore java.io.IOException
          	
          
          Bug type DE_MIGHT_IGNORE (click for details)
          In class org.apache.hadoop.hbase.replication.regionserver.ReplicationSource
          In method org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.processEndOfFile()
          Exception class java.io.IOException
          At ReplicationSource.java:[line 736]
          At ReplicationSource.java:[line 736]
          
          	try {
                    FileStatus stat = this.fs.getFileStatus(this.currentPath);
                    filesize = stat.getLen()+"";
                  } catch (IOException ex) {} <--- error
                  LOG.trace("Reached the end of a log, stats: " + getStats() +
                      ", and the length of the file is " + filesize);
          

          I will do some search first, and open separated Jira for above two issues if jira hasn't been opened yet.

          Demai

          Show
          Demai Ni added a comment - I went through the two -1 and didn't find the relationship with this patch for -1 javadoc. The javadoc tool appears to have generated 1 warning messages. Besides the existing (and known) Bytes.java about sun.misc.Unsafe, there is another one src/main/java/org/apache/hadoop/hbase/mapreduce/LabelExpander.java:188: warning - @ return tag has no arguments for -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. I also didn't find anything associated with this patch. But there is one related with replicationsource.java org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.processEndOfFile() might ignore java.io.IOException Bug type DE_MIGHT_IGNORE (click for details) In class org.apache.hadoop.hbase.replication.regionserver.ReplicationSource In method org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.processEndOfFile() Exception class java.io.IOException At ReplicationSource.java:[line 736] At ReplicationSource.java:[line 736] try { FileStatus stat = this .fs.getFileStatus( this .currentPath); filesize = stat.getLen()+""; } catch (IOException ex) {} <--- error LOG.trace( "Reached the end of a log, stats: " + getStats() + ", and the length of the file is " + filesize); I will do some search first, and open separated Jira for above two issues if jira hasn't been opened yet. Demai
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12617306/HBASE-9047-trunk-v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617306/HBASE-9047-trunk-v5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8068//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          thanks for the review.

          Show
          Demai Ni added a comment - thanks for the review.
          Hide
          Demai Ni added a comment -

          Lars Hofhansl, thank you so much. I will upload another patch (V5) in a couple minutes. There are some changes about ServerName in the past few weeks, so v5.patch changed this one-line

                return ServerName.valueOf(hostname, 1234, 1L);
          
          Show
          Demai Ni added a comment - Lars Hofhansl , thank you so much. I will upload another patch (V5) in a couple minutes. There are some changes about ServerName in the past few weeks, so v5.patch changed this one-line return ServerName.valueOf(hostname, 1234, 1L);
          Hide
          Lars Hofhansl added a comment -

          Oh... Didn't notice you were waiting for a review. I'll take a look today (or at very latest tomorrow). I agree this is pretty low risk and quite useful.

          Show
          Lars Hofhansl added a comment - Oh... Didn't notice you were waiting for a review. I'll take a look today (or at very latest tomorrow). I agree this is pretty low risk and quite useful.
          Hide
          Demai Ni added a comment -

          stack,Jean-Daniel Cryans,Lars Hofhansl,Ted Yu, wondering whether I can get some help to review this patch? The regression risk is quite low. For the mainstream code, I added a getter in ReplicationSourceManager, and add one more condition in the error patch of ReplicationSource. Most of the new code is in the standlone file : ReplicationSyncUp,java, and its UT TestReplicationSyncUpTool.java. Many thanks for your comments in advance.

          Show
          Demai Ni added a comment - stack , Jean-Daniel Cryans , Lars Hofhansl , Ted Yu , wondering whether I can get some help to review this patch? The regression risk is quite low. For the mainstream code, I added a getter in ReplicationSourceManager, and add one more condition in the error patch of ReplicationSource. Most of the new code is in the standlone file : ReplicationSyncUp,java, and its UT TestReplicationSyncUpTool.java. Many thanks for your comments in advance.
          Hide
          Demai Ni added a comment -

          Jean-Daniel Cryans,

          could you please review the new patch again? thanks

          the patch looks clean this time. the javadoc is unrelated (logged in HBASE-9916). I am still not sure about the 'mvn site goal to fail' though.

          Demai

          Show
          Demai Ni added a comment - Jean-Daniel Cryans , could you please review the new patch again? thanks the patch looks clean this time. the javadoc is unrelated (logged in HBASE-9916 ). I am still not sure about the 'mvn site goal to fail' though. Demai
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12612659/HBASE-9047-trunk-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612659/HBASE-9047-trunk-v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7779//console This message is automatically generated.
          Hide
          stack added a comment -

          Retry

          Show
          stack added a comment - Retry
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12612459/HBASE-9047-trunk-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612459/HBASE-9047-trunk-v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:488) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7762//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          new patch remove the 30 sec timeout at the end because the resourcemanager.oldsource checking is good enough to indicate no edits in queue.

          Show
          Demai Ni added a comment - new patch remove the 30 sec timeout at the end because the resourcemanager.oldsource checking is good enough to indicate no edits in queue.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12611148/HBASE-9047-trunk-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12611148/HBASE-9047-trunk-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7679//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          the failure from previous Hadoop QA is the same as HBASE-9863 intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs

          Show
          Demai Ni added a comment - the failure from previous Hadoop QA is the same as HBASE-9863 intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs
          Hide
          Demai Ni added a comment -

          update v3 for another Hadoop QA testing. v3 fixed the dodgy warnings about the static variable written from instance method

          Show
          Demai Ni added a comment - update v3 for another Hadoop QA testing. v3 fixed the dodgy warnings about the static variable written from instance method
          Hide
          Demai Ni added a comment -

          I am looking at the error returned from Hadoop QA.

          The failed testcase TestZookeeper failed at admin.createTable(...) , I couldn't figure out the relationship with my patch, and I am also able to run it locally on my machine. Another interesting finding is that https://builds.apache.org/job/PreCommit-HBASE-Build/7671/changes shows another patch Hbase-9221, so maybe the failure is caused by a different code.

          the Findbugs warnings(https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html) show some problems in my code. I will fix them.

          Demai

          Show
          Demai Ni added a comment - I am looking at the error returned from Hadoop QA. The failed testcase TestZookeeper failed at admin.createTable(...) , I couldn't figure out the relationship with my patch, and I am also able to run it locally on my machine. Another interesting finding is that https://builds.apache.org/job/PreCommit-HBASE-Build/7671/changes shows another patch Hbase-9221, so maybe the failure is caused by a different code. the Findbugs warnings( https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html ) show some problems in my code. I will fix them. Demai
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12610459/HBASE-9047-trunk-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12610459/HBASE-9047-trunk-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7671//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          update v2. This version fixed two bugs:
          1) in ReplicationSource.java, fix a situation that when HBase is crashed before flush data from .logs to .oldlogs (for example, kill -9). The new code will look into .logs folder
          2) during Syncup progress, check the oldSource(was incorrectly checking sources) to determine whether all data is pushed to peers

          Show
          Demai Ni added a comment - update v2. This version fixed two bugs: 1) in ReplicationSource.java, fix a situation that when HBase is crashed before flush data from .logs to .oldlogs (for example, kill -9). The new code will look into .logs folder 2) during Syncup progress, check the oldSource(was incorrectly checking sources) to determine whether all data is pushed to peers
          Hide
          Demai Ni added a comment -

          uploaded v1 patch for trunk with changes suggested by review comments.

          about the testcase, I switched it to extend TestReplicatonBase, and also only use one peer. But I still used the original table names and use peer1 for the name of slave cluster, just to keep the logic clear.

          Appreciate the comments and suggestions.

          Show
          Demai Ni added a comment - uploaded v1 patch for trunk with changes suggested by review comments. about the testcase, I switched it to extend TestReplicatonBase, and also only use one peer. But I still used the original table names and use peer1 for the name of slave cluster, just to keep the logic clear. Appreciate the comments and suggestions.
          Hide
          Demai Ni added a comment -

          Jean-Daniel Cryans, I am working to address your comments. would you please elaborate a bit more about the suggestion on 'Thread.sleep()'?

          .., so if the manager doesn't start a source and only recovers them, it seems to me that you could loop and wait until getSources() returns 0....

          The replicationManager.getSources() seems return the number of peers. for example of the testcase, it always return 2 regardless whether the syncup is completed or not. I also tried manager.getHLogs(), which doesn't work either. Should I use the replicationsource.metrics.sizeOfLogQueue?(need a public getter for that).

          many thanks.

          Demai

          Show
          Demai Ni added a comment - Jean-Daniel Cryans , I am working to address your comments. would you please elaborate a bit more about the suggestion on 'Thread.sleep()'? .., so if the manager doesn't start a source and only recovers them, it seems to me that you could loop and wait until getSources() returns 0.... The replicationManager.getSources() seems return the number of peers. for example of the testcase, it always return 2 regardless whether the syncup is completed or not. I also tried manager.getHLogs(), which doesn't work either. Should I use the replicationsource.metrics.sizeOfLogQueue?(need a public getter for that). many thanks. Demai
          Hide
          Demai Ni added a comment -

          Jean-Daniel Cryans

          thank you so much for the review comments and suggestions. I will remove the 'system.out.println', fix the typo, and remove the 'copyright' line. Also I will remove 'conf.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true)', and change the testcase per your suggestion.

          about the Thread.sleep(30000), many thanks for pointing it out. It would be a bug in making. I will put some zookeeper checking with a timeout loop.

          Let me look into the 'tool' class. I assume to make it a Runnable class, and use run() method as the main body is the key here. Thanks for the suggestion.

          As for the code style, I am using eclipse and import a hbase_eclipse_formatter.xml(http://hbase.apache.org/book/ides.html), but I realized that I must miss something from the experience of this and past patch submission. Is it the right way to follow? Is there a style checking script that I can run before submit? Thanks

          Have a nice weekend

          Demai

          Show
          Demai Ni added a comment - Jean-Daniel Cryans thank you so much for the review comments and suggestions. I will remove the 'system.out.println', fix the typo, and remove the 'copyright' line. Also I will remove 'conf.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true)', and change the testcase per your suggestion. about the Thread.sleep(30000), many thanks for pointing it out. It would be a bug in making. I will put some zookeeper checking with a timeout loop. Let me look into the 'tool' class. I assume to make it a Runnable class, and use run() method as the main body is the key here. Thanks for the suggestion. As for the code style, I am using eclipse and import a hbase_eclipse_formatter.xml( http://hbase.apache.org/book/ides.html ), but I realized that I must miss something from the experience of this and past patch submission. Is it the right way to follow? Is there a style checking script that I can run before submit? Thanks Have a nice weekend Demai
          Hide
          Jean-Daniel Cryans added a comment -

          Hi Demai,

          First, a friendly reminder to follow the coding standards as listed in this section: http://hbase.apache.org/book.html#developing

          Also, when writing a tool in HBase, we try to use the Tool class. See the tail of this file for an example: https://github.com/apache/hbase/blob/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java

          Specific comments on the patch:

          If you look at our other tools, we rarely give detailed logs on what it is doing. I'd just skip that one:

          +		System.out.println("getting config...");
          

          Are you sure we should turn on replication here? Our tools rarely turn on features:

          +		conf.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true);
          

          Typo:

          +		zkw = new ZooKeeperWatcher(conf, "syncupReplcation"
          

          I would skip the exclamation and all the dashes on the sides, same for the other messages. Also your message says "start" twice:

          +		System.out.println("------Start Replication Server start!------------");
          

          This sleep here I worry about. It's a very short time. Isn't there a better way to detect that we're done? In the description I was saying: " when starting ReplicationSourceManager you'd give it a fake region server ID, and you'd tell it not to start its own source.", so if the manager doesn't start a source and only recovers them, it seems to me that you could loop and wait until getSources() returns 0.

          +			Thread.sleep(30000);
          

          Regarding your test case, I don't think we need two peer, also it should extend TestReplicationBase which will setup a lot of infrastructure for your.

          I know that you copied code from another class, but new classes shouldn't have this line:

          + * Copyright 2011 The Apache Software Foundation
          

          Your method names shouldn't have underscores and they don't seem to convey what's really happening in them. For example, "testSyncUpTool_AfterDelete" does that plus it shuts down and restarts clusters. And what about this?

          +		// utilitySource.restartHBaseCluster(1);
          

          Thanks Demai.

          Show
          Jean-Daniel Cryans added a comment - Hi Demai, First, a friendly reminder to follow the coding standards as listed in this section: http://hbase.apache.org/book.html#developing Also, when writing a tool in HBase, we try to use the Tool class. See the tail of this file for an example: https://github.com/apache/hbase/blob/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java Specific comments on the patch: If you look at our other tools, we rarely give detailed logs on what it is doing. I'd just skip that one: + System.out.println("getting config..."); Are you sure we should turn on replication here? Our tools rarely turn on features: + conf.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true); Typo: + zkw = new ZooKeeperWatcher(conf, "syncupReplcation" I would skip the exclamation and all the dashes on the sides, same for the other messages. Also your message says "start" twice: + System.out.println("------Start Replication Server start!------------"); This sleep here I worry about. It's a very short time. Isn't there a better way to detect that we're done? In the description I was saying: " when starting ReplicationSourceManager you'd give it a fake region server ID, and you'd tell it not to start its own source.", so if the manager doesn't start a source and only recovers them, it seems to me that you could loop and wait until getSources() returns 0. + Thread.sleep(30000); Regarding your test case, I don't think we need two peer, also it should extend TestReplicationBase which will setup a lot of infrastructure for your. I know that you copied code from another class, but new classes shouldn't have this line: + * Copyright 2011 The Apache Software Foundation Your method names shouldn't have underscores and they don't seem to convey what's really happening in them. For example, "testSyncUpTool_AfterDelete" does that plus it shuts down and restarts clusters. And what about this? + // utilitySource.restartHBaseCluster(1); Thanks Demai.
          Hide
          Demai Ni added a comment -

          also upload a patch for 0.94.9. it is identical to trunk version, except for file locations

          Show
          Demai Ni added a comment - also upload a patch for 0.94.9. it is identical to trunk version, except for file locations
          Hide
          Demai Ni added a comment -

          originally, I plan to package the tool(ReplicationSyncUp.java) under either hadoop.hbase.util or hadoop.hbase.tool package. However, there is a change of ReplicationSourceManager.init(). This method was public on 0.94, but changed to protected on trunk. So I repack the ReplicationSyncUp in the same package together with ReplicationSourceManager. The alternative is to change sourcemanger.init() back to public.

          I consider to build a new class by extending from ReplicationSourceManager, but feel it is a bit over-kill here.

          Show
          Demai Ni added a comment - originally, I plan to package the tool(ReplicationSyncUp.java) under either hadoop.hbase.util or hadoop.hbase.tool package. However, there is a change of ReplicationSourceManager.init(). This method was public on 0.94, but changed to protected on trunk. So I repack the ReplicationSyncUp in the same package together with ReplicationSourceManager. The alternative is to change sourcemanger.init() back to public. I consider to build a new class by extending from ReplicationSourceManager, but feel it is a bit over-kill here.
          Hide
          Demai Ni added a comment -

          uploaded trunk patch for review and HadoopQA

          Show
          Demai Ni added a comment - uploaded trunk patch for review and HadoopQA
          Hide
          Demai Ni added a comment -

          folks,

          May I ask for your suggestions on a couple questions? and all comments on this prototype code are greatly appreciated.

          first question, I put this standalone tool under "package org.apache.hadoop.hbase.util.hbck", so it can be run on cli using "$ hbase org.apache.hadoop.hbase.util.hbck.ReplicationSyncUp". is this the right place?

          2nd question, I am using HConnection#getZooKeeperWatcher(), which is deprecated. I asked the question for alternative route through 'dev list'. Thanks for the comments and suggestions there. Also I'd like to provide the whole picture here. Basically, I'd like to access the zookeeper while hbase is offline; then use a dummy server to grab all the /hbase/replication/rs info from the original zookeeper to the new dummy server; and a replicationManager will be inited and do its magic. So the question is how to get the original zookeeper info.

          sorry that the prototype code is kind of messy, so I summarize the code flow below for easier read

          /**
           * hbase org.apache.hadoop.hbase.util.hbck.ReplicationSyncUp
           */
          public class ReplicationSyncUp {
          
            public static void main(String[] args) throws Exception {
          ...
              conf = HBaseConfiguration.create();
              connection = HConnectionManager.getConnection(conf);
              // the method is deprecate on 0.94, but the replacement is not backported to 0.94.
              // so have to use it for now.
              zkw = connection.getZooKeeperWatcher();
          ...
              replicationZK = new ReplicationZookeeper(connection, conf, zkw);
          ...
              replication = new Replication(new DummyServer(), fs, logDir, oldLogDir);
              manager = replication.getReplicationManager();
          ....
              List<WALActionsListener> listeners = new ArrayList<WALActionsListener>();
              listeners.add(replication);
          ....
              manager.init(); // the magic happens here
          .... // grab the /hbase/replication/rs from original zookeeper
          .... // put under the dummy server
          .... // push the Edits to slave cluster
              
              // tear down 
              manager.join();
           }
           static class DummyServer implements Server {
              String hostname;
          
              DummyServer() {
                // an unique name in case the first run fails
                hostname = System.currentTimeMillis() + ".SyncUpTool.replication.org";
              }
            ...
            }
          }
          
          
          Show
          Demai Ni added a comment - folks, May I ask for your suggestions on a couple questions? and all comments on this prototype code are greatly appreciated. first question, I put this standalone tool under "package org.apache.hadoop.hbase.util.hbck", so it can be run on cli using "$ hbase org.apache.hadoop.hbase.util.hbck.ReplicationSyncUp". is this the right place? 2nd question, I am using HConnection#getZooKeeperWatcher(), which is deprecated. I asked the question for alternative route through 'dev list'. Thanks for the comments and suggestions there. Also I'd like to provide the whole picture here. Basically, I'd like to access the zookeeper while hbase is offline; then use a dummy server to grab all the /hbase/replication/rs info from the original zookeeper to the new dummy server; and a replicationManager will be inited and do its magic. So the question is how to get the original zookeeper info. sorry that the prototype code is kind of messy, so I summarize the code flow below for easier read /** * hbase org.apache.hadoop.hbase.util.hbck.ReplicationSyncUp */ public class ReplicationSyncUp { public static void main( String [] args) throws Exception { ... conf = HBaseConfiguration.create(); connection = HConnectionManager.getConnection(conf); // the method is deprecate on 0.94, but the replacement is not backported to 0.94. // so have to use it for now. zkw = connection.getZooKeeperWatcher(); ... replicationZK = new ReplicationZookeeper(connection, conf, zkw); ... replication = new Replication( new DummyServer(), fs, logDir, oldLogDir); manager = replication.getReplicationManager(); .... List<WALActionsListener> listeners = new ArrayList<WALActionsListener>(); listeners.add(replication); .... manager.init(); // the magic happens here .... // grab the /hbase/replication/rs from original zookeeper .... // put under the dummy server .... // push the Edits to slave cluster // tear down manager.join(); } static class DummyServer implements Server { String hostname; DummyServer() { // an unique name in case the first run fails hostname = System .currentTimeMillis() + ".SyncUpTool.replication.org" ; } ... } }
          Hide
          Demai Ni added a comment -

          uploaded the prototype(built and tested on 0.94.9) for suggestions, comments and a couple questions.

          The good news is that the protoptye works. I tested with two tables, replicatedd to at least two peers, with combination of put and deletes.

          while, there are still a few items to be address. for instance, a testcase to be completed for this. And I also have a couple questions(in the next comment section) for your suggestions.

          Show
          Demai Ni added a comment - uploaded the prototype(built and tested on 0.94.9) for suggestions, comments and a couple questions. The good news is that the protoptye works. I tested with two tables, replicatedd to at least two peers, with combination of put and deletes. while, there are still a few items to be address. for instance, a testcase to be completed for this. And I also have a couple questions(in the next comment section) for your suggestions.
          Hide
          stack added a comment -

          Moving new feature out of 0.95/0.96. If done in time, we can bring it in... especially if external tool.

          Show
          stack added a comment - Moving new feature out of 0.95/0.96. If done in time, we can bring it in... especially if external tool.
          Hide
          Lars Hofhansl added a comment -

          Heh, just have to wait long enough.
          Seriously, thanks Demai!

          Show
          Lars Hofhansl added a comment - Heh, just have to wait long enough. Seriously, thanks Demai!
          Hide
          Demai Ni added a comment -

          JD and Lars, I will give a shoot at this one. JD, thanks for the suggestions.

          Show
          Demai Ni added a comment - JD and Lars, I will give a shoot at this one. JD, thanks for the suggestions.
          Hide
          Demai Ni added a comment -

          0.94 will be great(for me at least). Since I am looking for this feature for my customer, who will be on 0.94.

          Show
          Demai Ni added a comment - 0.94 will be great(for me at least). Since I am looking for this feature for my customer, who will be on 0.94.
          Hide
          Jean-Daniel Cryans added a comment -

          Are you volunteering?

          I think we can but I'm not sure exactly how it's gonna turn out. There's refactoring we have to do for sure since we're accessing ReplicationSourceManager outside of a region server.

          Show
          Jean-Daniel Cryans added a comment - Are you volunteering? I think we can but I'm not sure exactly how it's gonna turn out. There's refactoring we have to do for sure since we're accessing ReplicationSourceManager outside of a region server.
          Hide
          Lars Hofhansl added a comment -

          Can I add 0.94 to this as well?

          Show
          Lars Hofhansl added a comment - Can I add 0.94 to this as well?

            People

            • Assignee:
              Demai Ni
              Reporter:
              Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development