HBase
  1. HBase
  2. HBASE-9278

Reading Pre-namespace meta table edits kills the reader

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.98.0, 0.96.0
    • Component/s: migration, wal
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In upgrading to 0.96, there might be some meta/root table edits. Currently, we are just killing SplitLogWorker thread in case it sees any META, or ROOT waledit, which blocks log splitting/replaying of remaining WALs.

      2013-08-20 15:45:16,998 ERROR regionserver.SplitLogWorker (SplitLogWorker.java:run(210)) - unexpected error 
      java.lang.IllegalArgumentException: .META. no longer exists. The table has been renamed to hbase:meta
              at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:269)
              at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:261)
              at org.apache.hadoop.hbase.regionserver.wal.HLogKey.readFields(HLogKey.java:338)
              at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1898)
              at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1938)
              at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.readNext(SequenceFileLogReader.java:215)
              at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:98)
              at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
              at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getNextLogLine(HLogSplitter.java:582)
              at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:292)
              at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:209)
              at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:138)
              at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:358)
              at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:245)
              at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:205)
              at java.lang.Thread.run(Thread.java:662)
      2013-08-20 15:45:16,999 INFO  regionserver.SplitLogWorker (SplitLogWorker.java:run(212)) - SplitLogWorker localhost,60020,1377035111898 exiting
      
      1. HBase-9278-v0.patch
        7 kB
        Himanshu Vashishtha
      2. HBase-9278-v1.patch
        13 kB
        Himanshu Vashishtha
      3. HBase-9278-v1-1.patch
        13 kB
        Himanshu Vashishtha
      4. HBase-9278-v2.patch
        13 kB
        Himanshu Vashishtha
      5. HBase-9278-v2-rebase.patch
        13 kB
        Himanshu Vashishtha
      6. 9278v3.txt
        7 kB
        stack

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #700 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/700/)
          HBASE-9278 Reading Pre-namespace meta table edits kills the reader; COMMIT CONTINUATION (stack: rev 1518402)

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java
            HBASE-9278 Reading Pre-namespace meta table edits kills the reader (stack: rev 1518397)
          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #700 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/700/ ) HBASE-9278 Reading Pre-namespace meta table edits kills the reader; COMMIT CONTINUATION (stack: rev 1518402) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java HBASE-9278 Reading Pre-namespace meta table edits kills the reader (stack: rev 1518397) /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK #4442 (See https://builds.apache.org/job/HBase-TRUNK/4442/)
          HBASE-9278 Reading Pre-namespace meta table edits kills the reader; COMMIT CONTINUATION (stack: rev 1518402)

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java
            HBASE-9278 Reading Pre-namespace meta table edits kills the reader (stack: rev 1518397)
          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4442 (See https://builds.apache.org/job/HBase-TRUNK/4442/ ) HBASE-9278 Reading Pre-namespace meta table edits kills the reader; COMMIT CONTINUATION (stack: rev 1518402) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java HBASE-9278 Reading Pre-namespace meta table edits kills the reader (stack: rev 1518397) /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in hbase-0.95-on-hadoop2 #276 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/276/)
          HBASE-9278 Reading Pre-namespace meta table edits kills the reader; COMMIT CONTINUATION (stack: rev 1518403)

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java
            HBASE-9278 Reading Pre-namespace meta table edits kills the reader (stack: rev 1518398)
          • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          Show
          Hudson added a comment - SUCCESS: Integrated in hbase-0.95-on-hadoop2 #276 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/276/ ) HBASE-9278 Reading Pre-namespace meta table edits kills the reader; COMMIT CONTINUATION (stack: rev 1518403) /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java HBASE-9278 Reading Pre-namespace meta table edits kills the reader (stack: rev 1518398) /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in hbase-0.95 #501 (See https://builds.apache.org/job/hbase-0.95/501/)
          HBASE-9278 Reading Pre-namespace meta table edits kills the reader; COMMIT CONTINUATION (stack: rev 1518403)

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java
            HBASE-9278 Reading Pre-namespace meta table edits kills the reader (stack: rev 1518398)
          • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          Show
          Hudson added a comment - SUCCESS: Integrated in hbase-0.95 #501 (See https://builds.apache.org/job/hbase-0.95/501/ ) HBASE-9278 Reading Pre-namespace meta table edits kills the reader; COMMIT CONTINUATION (stack: rev 1518403) /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java HBASE-9278 Reading Pre-namespace meta table edits kills the reader (stack: rev 1518398) /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          Hide
          stack added a comment -

          Thanks. I added the test in another commit.

          Show
          stack added a comment - Thanks. I added the test in another commit.
          Hide
          Himanshu Vashishtha added a comment -

          Yeah, it needed a rebase post 7709. I attached the patch few secs ago. Just one nit: you missed the Test class.

          Show
          Himanshu Vashishtha added a comment - Yeah, it needed a rebase post 7709. I attached the patch few secs ago. Just one nit: you missed the Test class.
          Hide
          stack added a comment -

          Committed to 0.95 and trunk (my patch is what Himanshu posted but a comment on end of a method threw off application of his patch)

          Show
          stack added a comment - Committed to 0.95 and trunk (my patch is what Himanshu posted but a comment on end of a method threw off application of his patch)
          Hide
          stack added a comment -

          What I applied.

          Show
          stack added a comment - What I applied.
          Hide
          Himanshu Vashishtha added a comment -

          Rebased.

          Show
          Himanshu Vashishtha added a comment - Rebased.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12600462/HBase-9278-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6954//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600462/HBase-9278-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6954//console This message is automatically generated.
          Hide
          Himanshu Vashishtha added a comment -

          Attached patch takes incorporating Stack's reviews. Thanks.

          Show
          Himanshu Vashishtha added a comment - Attached patch takes incorporating Stack's reviews. Thanks.
          Hide
          Himanshu Vashishtha added a comment -

          Thanks for the review, Stack. Indeed, I don't need a mini cluster. Will remove it.

          Yes, we need to throw that iae so that we catch in the ReaderBase to call next() (we couldn't call next from HLogKey).

          Will upload a revised one now.

          Show
          Himanshu Vashishtha added a comment - Thanks for the review, Stack. Indeed, I don't need a mini cluster. Will remove it. Yes, we need to throw that iae so that we catch in the ReaderBase to call next() (we couldn't call next from HLogKey). Will upload a revised one now.
          Hide
          stack added a comment -

          Looks good. Why spin up a minitestcluster to add edits to a WAL?

          We throw the IAE if a ROOT edit? Is that right:

          + } else if (Bytes.toString(tablenameBytes).equals(TableName.OLD_ROOT_STR))

          { + this.tablename = TableName.OLD_ROOT_TABLE_NAME; + throw iae; + }

          else throw iae;

          These could get annoying I'd say especially if you are printing out full stack trace:

          + LOG.info("Got an old META edit, continuing with new format ", iae);

          Suggest not printing the IAE stack trace at least.

          I'll commit if you remove the test and address the above.

          Thanks H.

          Show
          stack added a comment - Looks good. Why spin up a minitestcluster to add edits to a WAL? We throw the IAE if a ROOT edit? Is that right: + } else if (Bytes.toString(tablenameBytes).equals(TableName.OLD_ROOT_STR)) { + this.tablename = TableName.OLD_ROOT_TABLE_NAME; + throw iae; + } else throw iae; These could get annoying I'd say especially if you are printing out full stack trace: + LOG.info("Got an old META edit, continuing with new format ", iae); Suggest not printing the IAE stack trace at least. I'll commit if you remove the test and address the above. Thanks H.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12600088/HBase-9278-v1-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600088/HBase-9278-v1-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6920//console This message is automatically generated.
          Hide
          Himanshu Vashishtha added a comment -

          Pretty sure TestFullLogReconstruction failure is not related. Re-attaching to let qa run again.

          Show
          Himanshu Vashishtha added a comment - Pretty sure TestFullLogReconstruction failure is not related. Re-attaching to let qa run again.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12600042/HBase-9278-v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestFullLogReconstruction

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600042/HBase-9278-v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.TestFullLogReconstruction Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6907//console This message is automatically generated.
          Hide
          Himanshu Vashishtha added a comment -

          Attached patch adds a test for reading old ROOT, and META edits. I had to add a new test class as it required SequenceFileLog/Writer(Reader) to write/read the old entries, in order to simulate reading 0.94.x wal files..

          Show
          Himanshu Vashishtha added a comment - Attached patch adds a test for reading old ROOT, and META edits. I had to add a new test class as it required SequenceFileLog/Writer(Reader) to write/read the old entries, in order to simulate reading 0.94.x wal files..
          Hide
          Himanshu Vashishtha added a comment -

          Okay. Let me try to come up with a test.

          Show
          Himanshu Vashishtha added a comment - Okay. Let me try to come up with a test.
          Hide
          stack added a comment -

          So far so good. A test would be ok in here though I'd say... an old wal edit w/ root/meta in it and ensure that new one has new tablename refs.

          Show
          stack added a comment - So far so good. A test would be ok in here though I'd say... an old wal edit w/ root/meta in it and ensure that new one has new tablename refs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12599749/HBase-9278-v0.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings).

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12599749/HBase-9278-v0.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings). +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6875//console This message is automatically generated.
          Hide
          Himanshu Vashishtha added a comment -

          Re-attaching the patch (was wrong one earlier)

          Show
          Himanshu Vashishtha added a comment - Re-attaching the patch (was wrong one earlier)
          Hide
          Himanshu Vashishtha added a comment -

          Stack:

          ?). meta is assigned ahead of all regions but only if Jeffrey Zhong 's fancy log splitter is enabled (which should be rare), right?

          No. Meta is assigned ahead without distributed log replaying too. That's why DD created a separate meta wal.

          If it is coming from a 0.94.x with meta wal feature:

          It needs testing as HMaster looks at zk to know previous meta location and we have a fresh zk state; it looks like it might skip meta logs before opening the region (I will test this).

          In any case, if we flush meta before closing previous cluster, and ask users not to do any DDL in the proximity of upgrade, we will be safe.

          Show
          Himanshu Vashishtha added a comment - Stack: ?). meta is assigned ahead of all regions but only if Jeffrey Zhong 's fancy log splitter is enabled (which should be rare), right? No. Meta is assigned ahead without distributed log replaying too. That's why DD created a separate meta wal. If it is coming from a 0.94.x with meta wal feature: It needs testing as HMaster looks at zk to know previous meta location and we have a fresh zk state; it looks like it might skip meta logs before opening the region (I will test this). In any case, if we flush meta before closing previous cluster, and ask users not to do any DDL in the proximity of upgrade, we will be safe.
          Hide
          Himanshu Vashishtha added a comment -

          Attached patch which takes care of old meta/root edit. I tested it with migrating from 0.94.x to trunk. Running jenkins internally.

          Show
          Himanshu Vashishtha added a comment - Attached patch which takes care of old meta/root edit. I tested it with migrating from 0.94.x to trunk. Running jenkins internally.
          Hide
          stack added a comment -

          So, we should be good if we just do the .META. > hbase:meta change and skip any -ROOT edits (are there any ROOT edits in WAL?). meta is assigned ahead of all regions but only if Jeffrey Zhong 's fancy log splitter is enabled (which should be rare), right? If it is a 0.94 install w/ the special meta WAL, the .META. -> hbase:meta fix will work for this case too.

          So, as I see it, we do not have to ensure clean shutdown of the cluster before move to 0.96? (If so, good)

          Show
          stack added a comment - So, we should be good if we just do the .META. > hbase:meta change and skip any -ROOT edits (are there any ROOT edits in WAL?). meta is assigned ahead of all regions but only if Jeffrey Zhong 's fancy log splitter is enabled (which should be rare), right? If it is a 0.94 install w/ the special meta WAL, the .META. -> hbase:meta fix will work for this case too. So, as I see it, we do not have to ensure clean shutdown of the cluster before move to 0.96? (If so, good)
          Hide
          Francis Liu added a comment -

          This is still in brain storm stage and now I'm doing research to see if the benefits are worth the effort. Once I have a concrete idea, I'll log a JIRA on it. Basically, with this support, it's possible to cut the downtime to min(or zero) to lower the bar for existing 0.94 customers to migrate to 0.96.

          Let me know what you find. I'm interested in a zero downtime approach.

          Show
          Francis Liu added a comment - This is still in brain storm stage and now I'm doing research to see if the benefits are worth the effort. Once I have a concrete idea, I'll log a JIRA on it. Basically, with this support, it's possible to cut the downtime to min(or zero) to lower the bar for existing 0.94 customers to migrate to 0.96. Let me know what you find. I'm interested in a zero downtime approach.
          Hide
          Francis Liu added a comment -

          Yes, I am working on a patch which when it sees old .META. entry, sets the tablename of the HLogKey to the new Meta table. And, just ignore the old root entries.

          Sounds good.

          Show
          Francis Liu added a comment - Yes, I am working on a patch which when it sees old .META. entry, sets the tablename of the HLogKey to the new Meta table. And, just ignore the old root entries. Sounds good.
          Hide
          Jeffrey Zhong added a comment -

          needed to support 0.94->0.96 replication.

          This is still in brain storm stage and now I'm doing research to see if the benefits are worth the effort. Once I have a concrete idea, I'll log a JIRA on it. Basically, with this support, it's possible to cut the downtime to min(or zero) to lower the bar for existing 0.94 customers to migrate to 0.96.

          Show
          Jeffrey Zhong added a comment - needed to support 0.94->0.96 replication. This is still in brain storm stage and now I'm doing research to see if the benefits are worth the effort. Once I have a concrete idea, I'll log a JIRA on it. Basically, with this support, it's possible to cut the downtime to min(or zero) to lower the bar for existing 0.94 customers to migrate to 0.96.
          Hide
          Himanshu Vashishtha added a comment -

          Yes, I am working on a patch which when it sees old .META. entry, sets the tablename of the HLogKey to the new Meta table. And, just ignore the old root entries.

          needed to support 0.94->0.96 replication.

          How this can be supported? Both the clusters have to be upped to have the same major number.

          So we have a separate meta log and a different pool for opening meta regions.

          Separate meta log is there in later version of 0.94, and is disabled by default. So, in most cases, meta edits will be interleaved with other edits in a 0.94.x deployment. FWIW, meta edits are not shipped/replicated.

          Show
          Himanshu Vashishtha added a comment - Yes, I am working on a patch which when it sees old .META. entry, sets the tablename of the HLogKey to the new Meta table. And, just ignore the old root entries. needed to support 0.94->0.96 replication. How this can be supported? Both the clusters have to be upped to have the same major number. So we have a separate meta log and a different pool for opening meta regions. Separate meta log is there in later version of 0.94, and is disabled by default. So, in most cases, meta edits will be interleaved with other edits in a 0.94.x deployment. FWIW, meta edits are not shipped/replicated.
          Hide
          Francis Liu added a comment -

          Are you going to make a patch Himanshu so we skip these meta and root edits?

          Instead of dropping we can hardwire the edits to '.META.' to go into 'hbase:meta'. Jeffrey Zhong and I were talking about this during meetup as one of the pieces needed to support 0.94->0.96 replication.

          We might be able to add to the upgrade a step where you ensure at a minimum the server carrying meta goes down clean?

          So we have a separate meta log and a different pool for opening meta regions. Is there such a priority when it comes to replication? I'm just thinking if it'd be cumbersome for users to wait until meta has been fully replicated before they can do the upgrade.

          Show
          Francis Liu added a comment - Are you going to make a patch Himanshu so we skip these meta and root edits? Instead of dropping we can hardwire the edits to '.META.' to go into 'hbase:meta'. Jeffrey Zhong and I were talking about this during meetup as one of the pieces needed to support 0.94->0.96 replication. We might be able to add to the upgrade a step where you ensure at a minimum the server carrying meta goes down clean? So we have a separate meta log and a different pool for opening meta regions. Is there such a priority when it comes to replication? I'm just thinking if it'd be cumbersome for users to wait until meta has been fully replicated before they can do the upgrade.
          Hide
          stack added a comment -

          What Himanshu Vashishtha said.

          Are you going to make a patch Himanshu so we skip these meta and root edits? We might be able to add to the upgrade a step where you ensure at a minimum the server carrying meta goes down clean?

          Show
          stack added a comment - What Himanshu Vashishtha said. Are you going to make a patch Himanshu so we skip these meta and root edits? We might be able to add to the upgrade a step where you ensure at a minimum the server carrying meta goes down clean?
          Hide
          Himanshu Vashishtha added a comment -

          Not in all cases I think. A cluster with replication enabled, will most likely be reading old logs.

          Show
          Himanshu Vashishtha added a comment - Not in all cases I think. A cluster with replication enabled, will most likely be reading old logs.
          Hide
          Francis Liu added a comment -

          Would it be good enough to require a clean shutdown of the cluster prior to an upgrade (ie all the data flush and logs removed)?

          Show
          Francis Liu added a comment - Would it be good enough to require a clean shutdown of the cluster prior to an upgrade (ie all the data flush and logs removed)?

            People

            • Assignee:
              Himanshu Vashishtha
              Reporter:
              Himanshu Vashishtha
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development