HBase
  1. HBase
  2. HBASE-5589

Add of the offline call to the Master Interface

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.6, 0.92.0, 0.94.0, 0.95.2
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: hbck
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hbck from HBASE-5128 requires an offline method on the master to properly cleanup state during certain assignment repair operations. This will this method will be added to recent and older versions of HBase.

      1. hbase-5589-0.90.patch
        3 kB
        Jonathan Hsieh
      2. hbase-5589-0.92.patch
        3 kB
        Jonathan Hsieh
      3. hbase-5589.patch
        3 kB
        Jonathan Hsieh
      4. hbase-5588-0.94.patch
        3 kB
        Jonathan Hsieh

        Issue Links

          Activity

          Hide
          ramkrishna.s.vasudevan added a comment -

          @Jon
          Sorry.. i saw it in bugs section. Just now noticed its an improvement.

          Show
          ramkrishna.s.vasudevan added a comment - @Jon Sorry.. i saw it in bugs section. Just now noticed its an improvement.
          Hide
          Jonathan Hsieh added a comment -

          @Ram I believe I've fixed this. At least according to the subversion tab here on this jira, CHANGES.txt has been updated.

          Show
          Jonathan Hsieh added a comment - @Ram I believe I've fixed this. At least according to the subversion tab here on this jira, CHANGES.txt has been updated.
          Hide
          ramkrishna.s.vasudevan added a comment -

          I think the CHANGES.txt is not updated for 0.90.

          Show
          ramkrishna.s.vasudevan added a comment - I think the CHANGES.txt is not updated for 0.90.
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Jon
          0.90 getting compilation error. Can you pls check it? Thanks.

          Show
          ramkrishna.s.vasudevan added a comment - @Jon 0.90 getting compilation error. Can you pls check it? Thanks.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #145 (See https://builds.apache.org/job/HBase-TRUNK-security/145/)
          HBASE-5589 Add of the offline call to the Master Interface (Revision 1303519)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #145 (See https://builds.apache.org/job/HBase-TRUNK-security/145/ ) HBASE-5589 Add of the offline call to the Master Interface (Revision 1303519) Result = SUCCESS jmhsieh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #334 (See https://builds.apache.org/job/HBase-0.92/334/)
          HBASE-5589 Add of the offline call to the Master Interface (Revision 1303621)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #334 (See https://builds.apache.org/job/HBase-0.92/334/ ) HBASE-5589 Add of the offline call to the Master Interface (Revision 1303621) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #45 (See https://builds.apache.org/job/HBase-0.94/45/)
          HBASE-5589 Add of the offline call to the Master Interface (Revision 1303520)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #45 (See https://builds.apache.org/job/HBase-0.94/45/ ) HBASE-5589 Add of the offline call to the Master Interface (Revision 1303520) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2691 (See https://builds.apache.org/job/HBase-TRUNK/2691/)
          HBASE-5589 Add of the offline call to the Master Interface (Revision 1303519)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2691 (See https://builds.apache.org/job/HBase-TRUNK/2691/ ) HBASE-5589 Add of the offline call to the Master Interface (Revision 1303519) Result = SUCCESS jmhsieh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519274/hbase-5588-0.94.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1237//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1237//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1237//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519274/hbase-5588-0.94.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1237//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1237//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1237//console This message is automatically generated.
          Hide
          stack added a comment -

          +1 on latest patch. Thanks Jon.

          Show
          stack added a comment - +1 on latest patch. Thanks Jon.
          Hide
          Jonathan Hsieh added a comment -

          @Stack. Ok. Attached 0.94 version without the bump and with a slightly updated comment.

          Show
          Jonathan Hsieh added a comment - @Stack. Ok. Attached 0.94 version without the bump and with a slightly updated comment.
          Hide
          stack added a comment -

          The 0.90 and 0.92 versions do not bump rpc to maintain compatibility but notes in comments that has been done.

          You are going to bump the rpc version in 0.94. If so, please don't. We want to be able to do a rolling restart going from 0.92 to 0.94.

          Show
          stack added a comment - The 0.90 and 0.92 versions do not bump rpc to maintain compatibility but notes in comments that has been done. You are going to bump the rpc version in 0.94. If so, please don't. We want to be able to do a rolling restart going from 0.92 to 0.94.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519253/hbase-5589.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.coprocessor.TestMasterObserver
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1236//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1236//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1236//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519253/hbase-5589.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1236//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1236//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1236//console This message is automatically generated.
          Hide
          Jonathan Hsieh added a comment -

          hbase-5589.patch bump HMasterInterface RPC version and apply to 0.94 and trunk
          The 0.90 and 0.92 versions do not bump rpc to maintain compatibility but notes in comments that has been done.

          Show
          Jonathan Hsieh added a comment - hbase-5589.patch bump HMasterInterface RPC version and apply to 0.94 and trunk The 0.90 and 0.92 versions do not bump rpc to maintain compatibility but notes in comments that has been done.
          Hide
          stack added a comment -

          Good. Thanks.

          Show
          stack added a comment - Good. Thanks.
          Hide
          Jonathan Hsieh added a comment -

          @stack

          Checked some more – it is in the 0.2 and 0.1 branches as well.

          https://github.com/apache/hbase/blob/0.1/src/java/org/apache/hadoop/hbase/ipc/HbaseRPC.java#L81

          Show
          Jonathan Hsieh added a comment - @stack Checked some more – it is in the 0.2 and 0.1 branches as well. https://github.com/apache/hbase/blob/0.1/src/java/org/apache/hadoop/hbase/ipc/HbaseRPC.java#L81
          Hide
          stack added a comment -

          @Jon OK. Then the order in the Interface is a mythology, left over from another time (maybe) when even method names were codes (though I just went back to 0.19 hbase and even there methodNames are Strings....). Good stuff.

          Show
          stack added a comment - @Jon OK. Then the order in the Interface is a mythology, left over from another time (maybe) when even method names were codes (though I just went back to 0.19 hbase and even there methodNames are Strings....). Good stuff.
          Hide
          Jonathan Hsieh added a comment -

          And the calling side encodes method names as part of a Writable Invocation so the interface it seems that order in the interface doesn't matter.

          https://github.com/apache/hbase/blob/trunk/src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java#L99

          This looks to be the case in 0.90 as well.
          https://github.com/apache/hbase/blob/0.90/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java#L105

          Here's where I need advice. Should bump the protocol version in trunk/0.96/0.94 (HMasterInterface)? What should I do about the protocol version number released version to maintain compatibility – specifically 0.90.x (HBaseRPCProtocolVersion) or 0.92.x (HMasterInterface)? Keep it the same?

          Show
          Jonathan Hsieh added a comment - And the calling side encodes method names as part of a Writable Invocation so the interface it seems that order in the interface doesn't matter. https://github.com/apache/hbase/blob/trunk/src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java#L99 This looks to be the case in 0.90 as well. https://github.com/apache/hbase/blob/0.90/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java#L105 Here's where I need advice. Should bump the protocol version in trunk/0.96/0.94 (HMasterInterface)? What should I do about the protocol version number released version to maintain compatibility – specifically 0.90.x (HBaseRPCProtocolVersion) or 0.92.x (HMasterInterface)? Keep it the same?
          Hide
          Jonathan Hsieh added a comment -

          Looks like from here the server side invocation uses method names (not an index)
          https://github.com/apache/hbase/blob/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java#L322

          Show
          Jonathan Hsieh added a comment - Looks like from here the server side invocation uses method names (not an index) https://github.com/apache/hbase/blob/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java#L322
          Hide
          Jonathan Hsieh added a comment -

          I've tested a version of this at adds the offline method "in the middle" of the HMasterInterface and then setup a cluster with a patched master (check version hash on web port) and 5 unpatched region servers (check version hash on web port). Methods below offline included balancer toggles – I assumed that if order of declaration mattered the methods after the inserted method would have compatibility problems. The methods below were balancer() and balance_switch(bool). I fired up an shell with the prepatched version issued balancer commands. They succeeded. I fired up a shell with patch version and issued balancer commands. They also succeeded.

          This might be due to some reflection/concidence with jruby so I'm going to dig into the rpc code to verify as well.

          Show
          Jonathan Hsieh added a comment - I've tested a version of this at adds the offline method "in the middle" of the HMasterInterface and then setup a cluster with a patched master (check version hash on web port) and 5 unpatched region servers (check version hash on web port). Methods below offline included balancer toggles – I assumed that if order of declaration mattered the methods after the inserted method would have compatibility problems. The methods below were balancer() and balance_switch(bool). I fired up an shell with the prepatched version issued balancer commands. They succeeded. I fired up a shell with patch version and issued balancer commands. They also succeeded. This might be due to some reflection/concidence with jruby so I'm going to dig into the rpc code to verify as well.
          Hide
          Jonathan Hsieh added a comment - - edited

          Found HBASE-5603 while testing rolling restart on 0.92.2-SNAPSHOT.

          Show
          Jonathan Hsieh added a comment - - edited Found HBASE-5603 while testing rolling restart on 0.92.2-SNAPSHOT.
          Hide
          Jonathan Hsieh added a comment -

          Need to check to see if order matters for rolling upgrade.

          Show
          Jonathan Hsieh added a comment - Need to check to see if order matters for rolling upgrade.

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development