HBase
  1. HBase
  2. HBASE-6316

Confirm can upgrade to 0.96 from 0.94 by just stopping and restarting

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Over in HBASE-6294, LarsH says you have to currently clear zk to get a 0.96 to start over data written by a 0.94. Need to fix it so don't have to do this – that zk state left over gets auto-migrated.

      1. 6316v3.txt
        7 kB
        stack
      2. 6316v3.txt
        6 kB
        stack
      3. 6316v2.txt
        6 kB
        stack
      4. a6a6562b777440fd9c34885428f5cb61.21e75333ada3d5bafb34bb918f29576c
        0.0 kB
        stack
      5. 6316.txt
        3 kB
        stack

        Activity

        Hide
        stack added a comment -

        Trying it, there are a few issues. Here is one:

        2012-10-02 22:13:55,804 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: error telling master we are up
        com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
                at org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188)
                at $Proxy10.regionServerStartup(Unknown Source)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2009)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:848)
                at java.lang.Thread.run(Thread.java:680)
        Caused by: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
                at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
                at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
                at org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:187)
                ... 4 more
        Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1798)
        
                at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1300)
                at org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:178)
                ... 4 more
        

        Ugly. Not end of world.

        This seems longer than it should be:

        2012-10-02 22:13:57,095 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 1537 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
        2012-10-02 22:13:58,629 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 3071 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
        2012-10-02 22:13:58,808 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at pynchon,60961,1349241221417
        2012-10-02 22:13:58,808 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Connected to master at pynchon/192.168.1.74:60961
        2012-10-02 22:13:58,808 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at pynchon,60961,1349241221417 that we are up with port=60964, startcode=1349241229036
        

        This is broke on local filesystem:

        Caused by: java.io.IOException: mark/reset not supported
                at org.apache.hadoop.fs.FSInputChecker.reset(FSInputChecker.java:418)
                at java.io.FilterInputStream.reset(FilterInputStream.java:200)
                at org.apache.hadoop.hbase.io.Reference.parseFrom(Reference.java:233)
                at org.apache.hadoop.hbase.io.Reference.read(Reference.java:180)
                at org.apache.hadoop.hbase.regionserver.StoreFile.<init>(StoreFile.java:259)
                at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:415)
                at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:413)
        

        Thats enough for now

        Show
        stack added a comment - Trying it, there are a few issues. Here is one: 2012-10-02 22:13:55,804 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: error telling master we are up com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188) at $Proxy10.regionServerStartup(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2009) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:848) at java.lang. Thread .run( Thread .java:680) Caused by: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:187) ... 4 more Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1798) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1300) at org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:178) ... 4 more Ugly. Not end of world. This seems longer than it should be: 2012-10-02 22:13:57,095 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 1537 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2012-10-02 22:13:58,629 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 3071 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. 2012-10-02 22:13:58,808 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at pynchon,60961,1349241221417 2012-10-02 22:13:58,808 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Connected to master at pynchon/192.168.1.74:60961 2012-10-02 22:13:58,808 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at pynchon,60961,1349241221417 that we are up with port=60964, startcode=1349241229036 This is broke on local filesystem: Caused by: java.io.IOException: mark/reset not supported at org.apache.hadoop.fs.FSInputChecker.reset(FSInputChecker.java:418) at java.io.FilterInputStream.reset(FilterInputStream.java:200) at org.apache.hadoop.hbase.io.Reference.parseFrom(Reference.java:233) at org.apache.hadoop.hbase.io.Reference.read(Reference.java:180) at org.apache.hadoop.hbase.regionserver.StoreFile.<init>(StoreFile.java:259) at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:415) at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:413) Thats enough for now
        Hide
        stack added a comment -

        Here is a fix for the failed parse of Reference files.

        Show
        stack added a comment - Here is a fix for the failed parse of Reference files.
        Hide
        stack added a comment -

        Also get this when try to look at UIs:

        HTTP ERROR 500
        
        Problem accessing /rs-status. Reason:
        
            Unresolved compilation problems: 
        	The import org.apache.hadoop.hbase.tmpl cannot be resolved
        	RSStatusTmpl cannot be resolved to a type
        	RSStatusTmpl cannot be resolved to a type
        Caused by:
        
        java.lang.Error: Unresolved compilation problems: 
        	The import org.apache.hadoop.hbase.tmpl cannot be resolved
        	RSStatusTmpl cannot be resolved to a type
        	RSStatusTmpl cannot be resolved to a type
        
        	at org.apache.hadoop.hbase.regionserver.RSStatusServlet.<init>(RSStatusServlet.java:29)
                ...
        
        Show
        stack added a comment - Also get this when try to look at UIs: HTTP ERROR 500 Problem accessing /rs-status. Reason: Unresolved compilation problems: The import org.apache.hadoop.hbase.tmpl cannot be resolved RSStatusTmpl cannot be resolved to a type RSStatusTmpl cannot be resolved to a type Caused by: java.lang.Error: Unresolved compilation problems: The import org.apache.hadoop.hbase.tmpl cannot be resolved RSStatusTmpl cannot be resolved to a type RSStatusTmpl cannot be resolved to a type at org.apache.hadoop.hbase.regionserver.RSStatusServlet.<init>(RSStatusServlet.java:29) ...
        Hide
        stack added a comment -

        Hmm... trying again I don't get the 500 building on this machine.

        Show
        stack added a comment - Hmm... trying again I don't get the 500 building on this machine.
        Hide
        stack added a comment -

        A 0.94 Writable that I want to check in under test data.

        Show
        stack added a comment - A 0.94 Writable that I want to check in under test data.
        Hide
        stack added a comment -

        v2 adds a test.

        Show
        stack added a comment - v2 adds a test.
        Hide
        stack added a comment -

        I don't get the 500 code if I do a clean build.

        Logs need an edit but can do that in another issue.

        Show
        stack added a comment - I don't get the 500 code if I do a clean build. Logs need an edit but can do that in another issue.
        Hide
        stack added a comment -

        This patch seems enough to make 0.96 start over 0.94.

        Show
        stack added a comment - This patch seems enough to make 0.96 start over 0.94.
        Hide
        stack added a comment -

        Add test size annotation

        Show
        stack added a comment - Add test size annotation
        Hide
        stack added a comment -

        Can I get a review for this blocker 0.96 issue? Thanks.

        Show
        stack added a comment - Can I get a review for this blocker 0.96 issue? Thanks.
        Hide
        Ted Yu added a comment -
        +        return convert(FSProtos.Reference.parseFrom(in));
        +      } else {
        

        nit: keyword else can be omitted.

        You may want to submit for QA run after including a6a6562b777440fd9c34885428f5cb61.21e75333ada3d5bafb34bb918f29576c in the patch.

        Show
        Ted Yu added a comment - + return convert(FSProtos.Reference.parseFrom(in)); + } else { nit: keyword else can be omitted. You may want to submit for QA run after including a6a6562b777440fd9c34885428f5cb61.21e75333ada3d5bafb34bb918f29576c in the patch.
        Hide
        stack added a comment -

        Here is what I committed. Thanks for the review Ted. The else was in previous code but I fixed it (Added WARNING about return in middle of a method).

        Show
        stack added a comment - Here is what I committed. Thanks for the review Ted. The else was in previous code but I fixed it (Added WARNING about return in middle of a method).
        Hide
        stack added a comment -

        Committed to trunk

        Show
        stack added a comment - Committed to trunk
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #215 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/215/)
        HBASE-6316 Confirm can upgrade to 0.96 from 0.94 by just stopping and restarting (Revision 1396428)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java
        • /hbase/trunk/hbase-server/src/test/data/a6a6562b777440fd9c34885428f5cb61.21e75333ada3d5bafb34bb918f29576c
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestReference.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #215 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/215/ ) HBASE-6316 Confirm can upgrade to 0.96 from 0.94 by just stopping and restarting (Revision 1396428) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java /hbase/trunk/hbase-server/src/test/data/a6a6562b777440fd9c34885428f5cb61.21e75333ada3d5bafb34bb918f29576c /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestReference.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3439 (See https://builds.apache.org/job/HBase-TRUNK/3439/)
        HBASE-6316 Confirm can upgrade to 0.96 from 0.94 by just stopping and restarting (Revision 1396428)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java
        • /hbase/trunk/hbase-server/src/test/data/a6a6562b777440fd9c34885428f5cb61.21e75333ada3d5bafb34bb918f29576c
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestReference.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3439 (See https://builds.apache.org/job/HBase-TRUNK/3439/ ) HBASE-6316 Confirm can upgrade to 0.96 from 0.94 by just stopping and restarting (Revision 1396428) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java /hbase/trunk/hbase-server/src/test/data/a6a6562b777440fd9c34885428f5cb61.21e75333ada3d5bafb34bb918f29576c /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestReference.java
        Hide
        stack added a comment -

        Marking closed.

        Show
        stack added a comment - Marking closed.

          People

          • Assignee:
            stack
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development