HBase
  1. HBase
  2. HBASE-10278 Provide better write predictability
  3. HBASE-10378

Divide HLog interface into User and Implementor specific interfaces

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: wal
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      HBase internals for the write ahead log have been refactored. Advanced users of HBase should be aware of the following changes.

      Public Audience
        - The Admin API for asking a region server to roll WAL files has changed from a synchronous command that returns a set of regions the WAL implementation would like flushed into an asynchronous command that returns nothing. Older clients relying on the former behavior will still be able to interact with newer servers, but the response body will always contain an empty list of regions to flush.
        - The shell command "hlog_roll" has been deprecated. Operators should use the "wal_roll" command instead. This command is subject to the changes described above for the Admin API to roll WAL files.
        - The command for analyzing write ahead logs has been renamed from 'hlog' to 'wal'. The old usage is deprecated and will be removed in a future version.
        - Some utility methods in the HBaseTesetingUtility related to testing write-ahead-logs were changed in incompatible ways. No functionality has been removed, but method names and arguments have changed. See the HBaseTestingUtility javadoc for details.
        - The WALPlayer utility has deprecated the configuration keys used for advanced customization. Users should switch to the updated configuration keys. See the usage information on the WALPlayer tool for details.
        - The HLogInputFormat utility class for processing logs with MapReduce has been deprecated and will be removed in a future version. Users should switch to the WALInputFormat.
        - The labeling of server metrics on the region server status pages changed. Previously, the number of backing files for the write ahead log was labeled 'Num. HLog Files'. If you wish to see this statistic now, please look for the label 'Num. WAL Files.' If you rely on JMX for these metrics, their location has not changed.

      LimitedPrivate(COPROC) Audience, LimitedPrivate(PHOENIX)
        - The RegionObserver API has been updated. The changes are both binary and source backwards compatible for coprocessors that use the BaseRegionObserver class. For those that implement RegionObserver directly the changes are binary backwards compatible. Depending on the internals of future HBase versions, coprocessors using the deprecated API may not see all WAL related events. Users are strongly encouraged to update their use of the API; see the RegionObserver javadoc for details.
        - Classes related to reading WAL entries (ReaderBase, ProtobufLogReader, SequenceFileLogReader) have changed in a backwards incompatible way. Users who referenced HLog.Reader directly or HLog.Entry will have to update. These changes do not impact compatibility with extant wal files.
        - The WALObserver API has been updated. The changes are both binary and source backwards compatible for coprocessors that use the BaseWALObserver class. For those that implement WALObserver directly the changes are binary backwards compatible. Depending on the internals of future HBase versions, coprocessors using the deprecated API may not see all WAL related events. Users are strongly encouraged to update their use of the API; see the WALObserver javadoc for details.
       - The WALCoprocessorEnvironment has changed in a backwards incompatible way. WALObserver coprocessors that relied on retrieving an object representing the write ahead log instance will have to be updated.

      LimitedPrivate(REPLICATION) Audience
       - The WALEntryFilter API has changed in a backwards incompatible way. Implementers will have to be updated.
       - The ReplicationEndpoint.ReplicateContext API has changed in a backwards incompatible way. Implementers who use this interface will have to be updated. These changes do not impact wire compatibility for replicating between clusters.
       - The HLogKey API is deprecated in favor of the WALKey API. Additionally, the HLogKey API has changed in a backwards incompatible way by changing from implementing WriteableComparable<HLogKey> to implementing Writeable and Comparable<WALKey>.
      Show
      HBase internals for the write ahead log have been refactored. Advanced users of HBase should be aware of the following changes. Public Audience   - The Admin API for asking a region server to roll WAL files has changed from a synchronous command that returns a set of regions the WAL implementation would like flushed into an asynchronous command that returns nothing. Older clients relying on the former behavior will still be able to interact with newer servers, but the response body will always contain an empty list of regions to flush.   - The shell command "hlog_roll" has been deprecated. Operators should use the "wal_roll" command instead. This command is subject to the changes described above for the Admin API to roll WAL files.   - The command for analyzing write ahead logs has been renamed from 'hlog' to 'wal'. The old usage is deprecated and will be removed in a future version.   - Some utility methods in the HBaseTesetingUtility related to testing write-ahead-logs were changed in incompatible ways. No functionality has been removed, but method names and arguments have changed. See the HBaseTestingUtility javadoc for details.   - The WALPlayer utility has deprecated the configuration keys used for advanced customization. Users should switch to the updated configuration keys. See the usage information on the WALPlayer tool for details.   - The HLogInputFormat utility class for processing logs with MapReduce has been deprecated and will be removed in a future version. Users should switch to the WALInputFormat.   - The labeling of server metrics on the region server status pages changed. Previously, the number of backing files for the write ahead log was labeled 'Num. HLog Files'. If you wish to see this statistic now, please look for the label 'Num. WAL Files.' If you rely on JMX for these metrics, their location has not changed. LimitedPrivate(COPROC) Audience, LimitedPrivate(PHOENIX)   - The RegionObserver API has been updated. The changes are both binary and source backwards compatible for coprocessors that use the BaseRegionObserver class. For those that implement RegionObserver directly the changes are binary backwards compatible. Depending on the internals of future HBase versions, coprocessors using the deprecated API may not see all WAL related events. Users are strongly encouraged to update their use of the API; see the RegionObserver javadoc for details.   - Classes related to reading WAL entries (ReaderBase, ProtobufLogReader, SequenceFileLogReader) have changed in a backwards incompatible way. Users who referenced HLog.Reader directly or HLog.Entry will have to update. These changes do not impact compatibility with extant wal files.   - The WALObserver API has been updated. The changes are both binary and source backwards compatible for coprocessors that use the BaseWALObserver class. For those that implement WALObserver directly the changes are binary backwards compatible. Depending on the internals of future HBase versions, coprocessors using the deprecated API may not see all WAL related events. Users are strongly encouraged to update their use of the API; see the WALObserver javadoc for details.  - The WALCoprocessorEnvironment has changed in a backwards incompatible way. WALObserver coprocessors that relied on retrieving an object representing the write ahead log instance will have to be updated. LimitedPrivate(REPLICATION) Audience  - The WALEntryFilter API has changed in a backwards incompatible way. Implementers will have to be updated.  - The ReplicationEndpoint.ReplicateContext API has changed in a backwards incompatible way. Implementers who use this interface will have to be updated. These changes do not impact wire compatibility for replicating between clusters.  - The HLogKey API is deprecated in favor of the WALKey API. Additionally, the HLogKey API has changed in a backwards incompatible way by changing from implementing WriteableComparable<HLogKey> to implementing Writeable and Comparable<WALKey>.

      Description

      HBASE-5937 introduces the HLog interface as a first step to support multiple WAL implementations. This interface is a good start, but has some limitations/drawbacks in its current state, such as:
      1) There is no clear distinction b/w User and Implementor APIs, and it provides APIs both for WAL users (append, sync, etc) and also WAL implementors (Reader/Writer interfaces, etc). There are APIs which are very much implementation specific (getFileNum, etc) and a user such as a RegionServer shouldn't know about it.
      2) There are about 14 methods in FSHLog which are not present in HLog interface but are used at several places in the unit test code. These tests typecast HLog to FSHLog, which makes it very difficult to test multiple WAL implementations without doing some ugly checks.

      I'd like to propose some changes in HLog interface that would ease the multi WAL story:

      1) Have two interfaces WAL and WALService. WAL provides APIs for implementors. WALService provides APIs for users (such as RegionServer).
      2) A skeleton implementation of the above two interface as the base class for other WAL implementations (AbstractWAL). It provides required fields for all subclasses (fs, conf, log dir, etc). Make a minimal set of test only methods and add this set in AbstractWAL.
      3) HLogFactory returns a WALService reference when creating a WAL instance; if a user need to access impl specific APIs (there are unit tests which get WAL from a HRegionServer and then call impl specific APIs), use AbstractWAL type casting,
      4) Make TestHLog abstract and let all implementors provide their respective test class which extends TestHLog (TestFSHLog, for example).

      1. HBASE-10378.addendum.patch
        2 kB
        ramkrishna.s.vasudevan
      2. HBASE-10378.5.patch.txt
        1.40 MB
        Sean Busbey
      3. HBASE-10378.4.patch.txt
        1.40 MB
        Sean Busbey
      4. HBASE-10378.3.patch.txt
        1.39 MB
        Sean Busbey
      5. 10378-2.patch
        288 kB
        Himanshu Vashishtha
      6. 10378-1.patch
        288 kB
        Himanshu Vashishtha

        Issue Links

          Activity

          Hide
          Sean Busbey added a comment -

          yay! closed.

          Show
          Sean Busbey added a comment - yay! closed.
          Hide
          Andrew Purtell added a comment -

          Yes, thanks, the issue is fixed

          Show
          Andrew Purtell added a comment - Yes, thanks, the issue is fixed
          Hide
          Andrew Purtell added a comment -

          Sorry, let me try now

          Show
          Andrew Purtell added a comment - Sorry, let me try now
          Hide
          Sean Busbey added a comment -

          waiting to make sure the fix in HBASE-12532 fixes the issue Andrew Purtell hit.

          Show
          Sean Busbey added a comment - waiting to make sure the fix in HBASE-12532 fixes the issue Andrew Purtell hit.
          Hide
          Lars Hofhansl added a comment -

          This is done, right? Can we close it?

          Show
          Lars Hofhansl added a comment - This is done, right? Can we close it?
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5796 (See https://builds.apache.org/job/HBase-TRUNK/5796/)
          HBASE-12532 TestFilter failing occasionally with ExitCodeException doing chmod since HBASE-10378 ADDENDUM (stack: rev 640274d5e1e4aa9b37689c64b3fffb2b52f92d87)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5796 (See https://builds.apache.org/job/HBase-TRUNK/5796/ ) HBASE-12532 TestFilter failing occasionally with ExitCodeException doing chmod since HBASE-10378 ADDENDUM (stack: rev 640274d5e1e4aa9b37689c64b3fffb2b52f92d87) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5793 (See https://builds.apache.org/job/HBase-TRUNK/5793/)
          HBASE-12532 TestFilter failing occasionally with ExitCodeException doing chmod since HBASE-10378 (stack: rev d87c550b948c11992660015ef26954a699929869)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5793 (See https://builds.apache.org/job/HBase-TRUNK/5793/ ) HBASE-12532 TestFilter failing occasionally with ExitCodeException doing chmod since HBASE-10378 (stack: rev d87c550b948c11992660015ef26954a699929869) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          ramkrishna.s.vasudevan added a comment -

          Stack
          No problem at all

          Show
          ramkrishna.s.vasudevan added a comment - Stack No problem at all
          Hide
          stack added a comment -

          Its fixed ramkrishna vasudevan My fault. I've been backing out stuff trying to get us stable again (made it worse!)

          Show
          stack added a comment - Its fixed ramkrishna vasudevan My fault. I've been backing out stuff trying to get us stable again (made it worse!)
          Hide
          ramkrishna.s.vasudevan added a comment -

          Adding an addendum.

          Show
          ramkrishna.s.vasudevan added a comment - Adding an addendum.
          Hide
          ramkrishna.s.vasudevan added a comment -

          The commit has this

          // request a roll
           227     admin.rollWALWriter(cluster.getServerHoldingRegion(region.getTableDesc().getTableName(),
           228       region.getRegionName()));
          

          But the latest patch has this attached in JIRA has this.

          admin.rollWALWriter(cluster.getServerHoldingRegion(region.getRegionName()));
          

          Am I the only person to get this error?

          Show
          ramkrishna.s.vasudevan added a comment - The commit has this // request a roll 227 admin.rollWALWriter(cluster.getServerHoldingRegion(region.getTableDesc().getTableName(), 228 region.getRegionName())); But the latest patch has this attached in JIRA has this. admin.rollWALWriter(cluster.getServerHoldingRegion(region.getRegionName())); Am I the only person to get this error?
          Hide
          Sean Busbey added a comment -

          Andrew Purtell with the patch from HBASE-12532 in place, TestHRegion in a loop is stable on my OSX box without mucking with anything. Can you give it a try?

          Show
          Sean Busbey added a comment - Andrew Purtell with the patch from HBASE-12532 in place, TestHRegion in a loop is stable on my OSX box without mucking with anything. Can you give it a try?
          Hide
          ramkrishna.s.vasudevan added a comment -

          On updation of the master branch I get the following error

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.
          2:testCompile (default-testCompile) on project hbase-server: Compilation failure
          
          [ERROR] /C:/HBase_git/hbase/hbase/hbase-server/src/test/java/org/apache/hadoop/h
          base/replication/TestMultiSlaveReplication.java:[227,32] no suitable method foun
          d for getServerHoldingRegion(org.apache.hadoop.hbase.TableName,byte[])
          [ERROR] method org.apache.hadoop.hbase.MiniHBaseCluster.getServerHoldingRegion(b
          yte[]) is not applicable
          [ERROR] (actual and formal argument lists differ in length)
          [ERROR] -> [Help 1]
          
          Show
          ramkrishna.s.vasudevan added a comment - On updation of the master branch I get the following error [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3. 2:testCompile ( default -testCompile) on project hbase-server: Compilation failure [ERROR] /C:/HBase_git/hbase/hbase/hbase-server/src/test/java/org/apache/hadoop/h base/replication/TestMultiSlaveReplication.java:[227,32] no suitable method foun d for getServerHoldingRegion(org.apache.hadoop.hbase.TableName, byte []) [ERROR] method org.apache.hadoop.hbase.MiniHBaseCluster.getServerHoldingRegion(b yte[]) is not applicable [ERROR] (actual and formal argument lists differ in length) [ERROR] -> [Help 1]
          Hide
          ramkrishna.s.vasudevan added a comment -

          Thanks Sean Busbey. Will follow HBASE-5699. Added some comments in RB.

          Show
          ramkrishna.s.vasudevan added a comment - Thanks Sean Busbey . Will follow HBASE-5699 . Added some comments in RB.
          Hide
          Sean Busbey added a comment -

          That sounds like a problem I ran into when I rebased to current master on Friday. In my case it was one of the HBaseFsck tests that died a horrible death and got into that state. But it was the exact same thing with the mysterious "can't get writer" and splat.

          In my case the problem was that max open files was set to something ridiculous like 200. Can you check that on your macbook?

          Show
          Sean Busbey added a comment - That sounds like a problem I ran into when I rebased to current master on Friday. In my case it was one of the HBaseFsck tests that died a horrible death and got into that state. But it was the exact same thing with the mysterious "can't get writer" and splat. In my case the problem was that max open files was set to something ridiculous like 200. Can you check that on your macbook?
          Hide
          Andrew Purtell added a comment -

          No problems on a Linux system.

          Show
          Andrew Purtell added a comment - No problems on a Linux system.
          Hide
          Andrew Purtell added a comment -

          Pardon if I'm implicating the wrong issue.

          Working on a Macbook dev box.

          I have found that some TestHRegion unit tests fail now because the can't get a WAL writer:

          $ mvn -DskipTests clean install
          $ mvn test -Dtest=TestHRegion
          Tests in error: 
            TestHRegion.testDelete_multiDeleteColumn:1836->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer
            TestHRegion.testRegionReplicaSecondary:4587 » IOException cannot get log writer
            TestHRegion.testScanner_DeleteOneFamilyNotAnother:2096->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer
            TestHRegion.testScanner_JoinedScannersWithLimits:3156->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer
          

          For example:

          testRegionReplicaSecondary(org.apache.hadoop.hbase.regionserver.TestHRegion)  Time elapsed: 0.063 sec  <<< ERROR!
          java.io.IOException: cannot get log writer
          	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
          	at org.apache.hadoop.util.Shell.run(Shell.java:455)
          	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
          	at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
          	at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
          	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:646)
          	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:293)
          	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:451)
          	at org.apache.hadoop.fs.ChecksumFileSystem.createNonRecursive(ChecksumFileSystem.java:469)
          	at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1086)
          	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
          	at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:361)
          	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:736)
          	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:701)
          	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:612)
          	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:549)
          	at org.apache.hadoop.hbase.wal.DefaultWALProvider.init(DefaultWALProvider.java:97)
          	at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:139)
          	at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:171)
          	at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4566)
          	at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4533)
          	at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4508)
          	at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4585)
          	at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4469)
          	at org.apache.hadoop.hbase.regionserver.TestHRegion.testRegionReplicaSecondary(TestHRegion.java:4587)
          

          Then, I can't clean:

          $ mvn clean
          ...
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on project hbase-server: Failed to clean project: Failed to delete /Users/apurtell/src/hbase/hbase-server/target/test-data/c8c12cda-613a-488d-9a9c-2463c8f884d5/WALs -> [Help 1]
          

          I need to sudo rm -rf .

          It wasn't like this last week.

          Show
          Andrew Purtell added a comment - Pardon if I'm implicating the wrong issue. Working on a Macbook dev box. I have found that some TestHRegion unit tests fail now because the can't get a WAL writer: $ mvn -DskipTests clean install $ mvn test -Dtest=TestHRegion Tests in error: TestHRegion.testDelete_multiDeleteColumn:1836->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer TestHRegion.testRegionReplicaSecondary:4587 » IOException cannot get log writer TestHRegion.testScanner_DeleteOneFamilyNotAnother:2096->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer TestHRegion.testScanner_JoinedScannersWithLimits:3156->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer For example: testRegionReplicaSecondary(org.apache.hadoop.hbase.regionserver.TestHRegion) Time elapsed: 0.063 sec <<< ERROR! java.io.IOException: cannot get log writer at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.util.Shell.execCommand(Shell.java:791) at org.apache.hadoop.util.Shell.execCommand(Shell.java:774) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:646) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:293) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:451) at org.apache.hadoop.fs.ChecksumFileSystem.createNonRecursive(ChecksumFileSystem.java:469) at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1086) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90) at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:361) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:736) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:701) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:612) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:549) at org.apache.hadoop.hbase.wal.DefaultWALProvider.init(DefaultWALProvider.java:97) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:139) at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:171) at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4566) at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4533) at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4508) at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4585) at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4469) at org.apache.hadoop.hbase.regionserver.TestHRegion.testRegionReplicaSecondary(TestHRegion.java:4587) Then, I can't clean: $ mvn clean ... [ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on project hbase-server: Failed to clean project: Failed to delete /Users/apurtell/src/hbase/hbase-server/target/test-data/c8c12cda-613a-488d-9a9c-2463c8f884d5/WALs -> [Help 1] I need to sudo rm -rf . It wasn't like this last week.
          Hide
          Elliott Clark added a comment -

          Thanks Sean Busbey This is awesome. I actually held off on creating a feature branch so that I could get this patch in.

          Show
          Elliott Clark added a comment - Thanks Sean Busbey This is awesome. I actually held off on creating a feature branch so that I could get this patch in.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5787 (See https://builds.apache.org/job/HBase-TRUNK/5787/)
          HBASE-10378 Refactor write-ahead-log implementation – ADDEDNUM (stack: rev b94e6738d920cb5bf297214883e0cf29522597b9)

          • hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALSourceImpl.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/HLogSplitterHandler.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/Compressor.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFactory.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationHLogReaderManager.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationHLogReaderManager.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALCoprocessorEnvironment.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SecureProtobufLogWriter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClient.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
          • hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithEncryption.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedSequenceFileLogWriter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationKillMasterRSCompressed.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALObserver.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterFileSystem.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationChangingPeerRegionservers.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          • hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestMetricsHLogSource.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotLogCleaner.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALObserver.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ScopeWALEntryFilter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
          • hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
          • hbase-protocol/src/main/protobuf/RegionServerStatus.proto
          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSVisitor.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
          • hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon
          • hbase-shell/src/main/ruby/shell.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/DroppedSnapshotException.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • hbase-common/src/main/resources/hbase-default.xml
          • bin/hbase.cmd
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestSecureHLog.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/SplitLogWorkerCoordination.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerAccounting.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
          • src/main/docbkx/troubleshooting.xml
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java
          • src/main/docbkx/ops_mgt.xml
          • hbase-shell/src/main/ruby/shell/commands/hlog_roll.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/migration/NamespaceUpgrade.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ChainWALEntryFilter.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
          • hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplitCompressed.java
          • hbase-protocol/src/main/protobuf/ZooKeeper.proto
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java
          • hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
          • hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerCoprocessorRpcChannel.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseWALObserver.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestSecureWALReplay.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/SystemTableWALEntryFilter.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
          • bin/hbase
          • hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/HLogLink.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/SplitLogTask.java
          • hbase-protocol/src/main/protobuf/WAL.proto
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/WALEntryFilter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultySequenceFileLogReader.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
          • src/main/docbkx/performance.xml
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogReaderOnSecureHLog.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
          • hbase-protocol/src/main/protobuf/Admin.proto
          • hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterFileSystemSource.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWAL.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java
          • hbase-shell/src/main/ruby/hbase/admin.rb
          • hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollPeriod.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinorCompaction.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultyHLog.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
          • hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALSource.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WriterBase.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestInvocationRecordFilter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java
          • src/main/docbkx/book.xml
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetaLogRoller.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5787 (See https://builds.apache.org/job/HBase-TRUNK/5787/ ) HBASE-10378 Refactor write-ahead-log implementation – ADDEDNUM (stack: rev b94e6738d920cb5bf297214883e0cf29522597b9) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALSourceImpl.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/HLogSplitterHandler.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/Compressor.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFactory.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationHLogReaderManager.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationHLogReaderManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALCoprocessorEnvironment.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SecureProtobufLogWriter.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClient.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithEncryption.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedSequenceFileLogWriter.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationKillMasterRSCompressed.java hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALObserver.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterFileSystem.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationChangingPeerRegionservers.java hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestMetricsHLogSource.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotLogCleaner.java hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALObserver.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ScopeWALEntryFilter.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java hbase-protocol/src/main/protobuf/RegionServerStatus.proto hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSVisitor.java hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon hbase-shell/src/main/ruby/shell.rb hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java hbase-client/src/main/java/org/apache/hadoop/hbase/DroppedSnapshotException.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java hbase-common/src/main/resources/hbase-default.xml bin/hbase.cmd hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestSecureHLog.java hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/SplitLogWorkerCoordination.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerAccounting.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java src/main/docbkx/troubleshooting.xml hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java src/main/docbkx/ops_mgt.xml hbase-shell/src/main/ruby/shell/commands/hlog_roll.rb hbase-server/src/main/java/org/apache/hadoop/hbase/migration/NamespaceUpgrade.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ChainWALEntryFilter.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplitCompressed.java hbase-protocol/src/main/protobuf/ZooKeeper.proto hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerCoprocessorRpcChannel.java hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseWALObserver.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestSecureWALReplay.java hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/SystemTableWALEntryFilter.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java bin/hbase hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/HLogLink.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java hbase-server/src/main/java/org/apache/hadoop/hbase/SplitLogTask.java hbase-protocol/src/main/protobuf/WAL.proto hbase-server/src/main/java/org/apache/hadoop/hbase/replication/WALEntryFilter.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultySequenceFileLogReader.java hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java src/main/docbkx/performance.xml hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogReaderOnSecureHLog.java hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java hbase-protocol/src/main/protobuf/Admin.proto hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterFileSystemSource.java hbase-server/src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWAL.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java hbase-shell/src/main/ruby/hbase/admin.rb hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollPeriod.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinorCompaction.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultyHLog.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALSource.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WriterBase.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestInvocationRecordFilter.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java src/main/docbkx/book.xml hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetaLogRoller.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java
          Hide
          stack added a comment -

          Had to do an addendum... git am --continue failed pick up the deletes for me (pilot issue I'm sure). Here is addendum I added b94e6738d920cb5bf297214883e0cf29522597b9

          Show
          stack added a comment - Had to do an addendum... git am --continue failed pick up the deletes for me (pilot issue I'm sure). Here is addendum I added b94e6738d920cb5bf297214883e0cf29522597b9
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5786 (See https://builds.apache.org/job/HBase-TRUNK/5786/)
          HBASE-10378 Refactor write-ahead-log implementation (stack: rev f5e05eb836a807e418ebcf432fb0f481b40277ac)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestSecureWAL.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DisabledWALProvider.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DefaultWALProvider.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALRecordReader.java
          • hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestMetricsWALSource.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestProtobufLog.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALMethods.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationWALReaderManager.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestDefaultWALProviderWithHLogKey.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFiltering.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedLogWriter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/FaultyFSLog.java
          • hbase-shell/src/main/ruby/shell/commands/wal_roll.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALReaderOnSecureWAL.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestDefaultWALProvider.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFactory.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/WALSplitterHandler.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationWALReaderManager.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/io/WALLink.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALUtil.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALProvider.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5786 (See https://builds.apache.org/job/HBase-TRUNK/5786/ ) HBASE-10378 Refactor write-ahead-log implementation (stack: rev f5e05eb836a807e418ebcf432fb0f481b40277ac) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestSecureWAL.java hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DisabledWALProvider.java hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DefaultWALProvider.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALRecordReader.java hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestMetricsWALSource.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestProtobufLog.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALMethods.java hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationWALReaderManager.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestDefaultWALProviderWithHLogKey.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFiltering.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedLogWriter.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/FaultyFSLog.java hbase-shell/src/main/ruby/shell/commands/wal_roll.rb hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALReaderOnSecureWAL.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestDefaultWALProvider.java hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFactory.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/WALSplitterHandler.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationWALReaderManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/WALLink.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALProvider.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          stack added a comment -

          Pushed to master. Nice work Sean Busbey I think I got it all in.

          Show
          stack added a comment - Pushed to master. Nice work Sean Busbey I think I got it all in.
          Hide
          stack added a comment -

          Let me get this in before it rots. Pardon me if I appear overly-aggressive (I'm just being lazy). No problem backing it out if objection.

          Show
          stack added a comment - Let me get this in before it rots. Pardon me if I appear overly-aggressive (I'm just being lazy). No problem backing it out if objection.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12682182/HBASE-10378.5.patch.txt
          against trunk revision .
          ATTACHMENT ID: 12682182

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 349 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 checkstyle. The applied patch generated 3833 checkstyle errors (more than the trunk's current 3788 errors).

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          +set corecommands=shell master regionserver thrift thrift2 rest avro hlog wal hbck hfile zookeeper zkcli upgrade mapredcp
          + * 2010-06-15 11:52:28,511 WARN [DataStreamer for file /hbase/.logs/wal.1276627923013 block blk_928005470262850423_1021] hdfs.DFSClient$DFSOutputStream(2657): Error Recovery for block blk_928005470262850423_1021 failed because recovery from primary datanode 127.0.0.1:53683 failed 4 times. Pipeline was 127.0.0.1:53687, 127.0.0.1:53683. Will retry...
          + <screen langauge="bourne"> $ ./bin/hbase wal hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/10.10.21.10%3A60020.1283973724012</screen>
          + <screen langauge="bourne"> $ ./bin/hbase hlog hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/10.10.21.10%3A60020.1283973724012</screen>

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer.testImmediateAssignment(TestBaseLoadBalancer.java:136)
          at org.apache.sentry.tests.e2e.hive.TestMovingToProduction.testMovingTable2(TestMovingToProduction.java:183)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/checkstyle-aggregate.html

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12682182/HBASE-10378.5.patch.txt against trunk revision . ATTACHMENT ID: 12682182 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 349 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 checkstyle . The applied patch generated 3833 checkstyle errors (more than the trunk's current 3788 errors). +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: +set corecommands=shell master regionserver thrift thrift2 rest avro hlog wal hbck hfile zookeeper zkcli upgrade mapredcp + * 2010-06-15 11:52:28,511 WARN [DataStreamer for file /hbase/.logs/wal.1276627923013 block blk_928005470262850423_1021] hdfs.DFSClient$DFSOutputStream(2657): Error Recovery for block blk_928005470262850423_1021 failed because recovery from primary datanode 127.0.0.1:53683 failed 4 times. Pipeline was 127.0.0.1:53687, 127.0.0.1:53683. Will retry... + <screen langauge="bourne"> $ ./bin/hbase wal hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/10.10.21.10%3A60020.1283973724012</screen> + <screen langauge="bourne"> $ ./bin/hbase hlog hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/10.10.21.10%3A60020.1283973724012</screen> +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 2 zombie test(s): at org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer.testImmediateAssignment(TestBaseLoadBalancer.java:136) at org.apache.sentry.tests.e2e.hive.TestMovingToProduction.testMovingTable2(TestMovingToProduction.java:183) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11734//console This message is automatically generated.
          Hide
          Sean Busbey added a comment -

          The initial implementation done in this refactoring only replicates the behavior from prior to the patch; i.e. one wal per region server.

          HBASE-5699 has the patch for building on this to do using multiple wals based on a grouping. You can evaluate that patch without most of the context here if you rely on the fact that on region open we request a WAL based on the region's encoded name. We should keep the discussion of how one would e.g. group by table on that ticket. (the short version is you'd have to look up a byte[] that represents the table).

          Show
          Sean Busbey added a comment - The initial implementation done in this refactoring only replicates the behavior from prior to the patch; i.e. one wal per region server. HBASE-5699 has the patch for building on this to do using multiple wals based on a grouping. You can evaluate that patch without most of the context here if you rely on the fact that on region open we request a WAL based on the region's encoded name. We should keep the discussion of how one would e.g. group by table on that ticket. (the short version is you'd have to look up a byte[] that represents the table).
          Hide
          ramkrishna.s.vasudevan added a comment -

          Still going thro the RB. One more day would be great. In case of open region - when there is a grouping done based on table (for eg), how are we handling the assignment of WAL to those regions specific to a table. I am still not there in the review but that would be interesting to see. I am in the 5th page now.

          Show
          ramkrishna.s.vasudevan added a comment - Still going thro the RB. One more day would be great. In case of open region - when there is a grouping done based on table (for eg), how are we handling the assignment of WAL to those regions specific to a table. I am still not there in the review but that would be interesting to see. I am in the 5th page now.
          Hide
          Sean Busbey added a comment -

          Patch fixes long lines except for those that rely on not wrapping for formatting.

          The remaining javadoc warning isn't related (will file follow on to fix it). Haven't worked out what's busted in checkstyle yet.

          Show
          Sean Busbey added a comment - Patch fixes long lines except for those that rely on not wrapping for formatting. The remaining javadoc warning isn't related (will file follow on to fix it). Haven't worked out what's busted in checkstyle yet.
          Hide
          stack added a comment -

          +1 on commit to master branch after reviewing up on rb. Will let this hang out until tomorrow in case any other opinions out there (see Sean Busbey release notes above for what changes)

          Show
          stack added a comment - +1 on commit to master branch after reviewing up on rb. Will let this hang out until tomorrow in case any other opinions out there (see Sean Busbey release notes above for what changes)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12681992/HBASE-10378.4.patch.txt
          against trunk revision .
          ATTACHMENT ID: 12681992

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 349 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          -1 checkstyle. The applied patch generated 3855 checkstyle errors (more than the trunk's current 3788 errors).

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          +set corecommands=shell master regionserver thrift thrift2 rest avro hlog wal hbck hfile zookeeper zkcli upgrade mapredcp
          + final WAL wal = walFactory.getMetaWAL(HRegionInfo.FIRST_META_REGIONINFO.getEncodedNameAsBytes());
          + LOG.warn("Exception attempting to fetch wal coprocessor information for the common wal; skipping.");
          + LOG.warn("Exception attempting to fetch wal coprocessor information for region " + region + "; skipping.");
          + LOG.error("Could not retrieve WAL information for region " + r.getRegionInfo() + "; not adding to moved regions.");
          + private final ConcurrentHashMap<WAL, Boolean> walNeedsRoll = new ConcurrentHashMap<WAL, Boolean>();
          + final byte [][] regionsToFlush = wal.rollWriter(periodic || entry.getValue().booleanValue());
          + // in the case of the null suffix, we need to make sure the filename ends with the timestamp.
          + return org.apache.commons.lang.StringUtils.isNumeric(fileNameString.substring(prefixPathStr.length()));
          + throw new IllegalArgumentException("The log file " + fileName + " doesn't belong to this wal. (" + toString() + ")");

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12681992/HBASE-10378.4.patch.txt against trunk revision . ATTACHMENT ID: 12681992 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 349 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. -1 checkstyle . The applied patch generated 3855 checkstyle errors (more than the trunk's current 3788 errors). +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: +set corecommands=shell master regionserver thrift thrift2 rest avro hlog wal hbck hfile zookeeper zkcli upgrade mapredcp + final WAL wal = walFactory.getMetaWAL(HRegionInfo.FIRST_META_REGIONINFO.getEncodedNameAsBytes()); + LOG.warn("Exception attempting to fetch wal coprocessor information for the common wal; skipping."); + LOG.warn("Exception attempting to fetch wal coprocessor information for region " + region + "; skipping."); + LOG.error("Could not retrieve WAL information for region " + r.getRegionInfo() + "; not adding to moved regions."); + private final ConcurrentHashMap<WAL, Boolean> walNeedsRoll = new ConcurrentHashMap<WAL, Boolean>(); + final byte [][] regionsToFlush = wal.rollWriter(periodic || entry.getValue().booleanValue()); + // in the case of the null suffix, we need to make sure the filename ends with the timestamp. + return org.apache.commons.lang.StringUtils.isNumeric(fileNameString.substring(prefixPathStr.length())); + throw new IllegalArgumentException("The log file " + fileName + " doesn't belong to this wal. (" + toString() + ")"); +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11717//console This message is automatically generated.
          Hide
          Sean Busbey added a comment -

          updated patch with most of the changed suggested by QA Bot.

          Didn't wrap the long line in hbase.cmd because

          • I can't tell if our Windows scripts are for PowerShell or earlier and line continuations are handled differently in those two cases
          • there are a ton of other lines > 100 columns in there already.
          Show
          Sean Busbey added a comment - updated patch with most of the changed suggested by QA Bot. Didn't wrap the long line in hbase.cmd because I can't tell if our Windows scripts are for PowerShell or earlier and line continuations are handled differently in those two cases there are a ton of other lines > 100 columns in there already.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12681773/HBASE-10378.3.patch.txt
          against trunk revision .
          ATTACHMENT ID: 12681773

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 349 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 25 warning messages.

          -1 checkstyle. The applied patch generated 3861 checkstyle errors (more than the trunk's current 3788 errors).

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          +set corecommands=shell master regionserver thrift thrift2 rest avro hlog wal hbck hfile zookeeper zkcli upgrade mapredcp
          + * Protocol buffer version of WALKey; see WALKey comment, not really a key but WALEdit header for some KVs
          + * Protocol buffer version of WALKey; see WALKey comment, not really a key but WALEdit header for some KVs
          +// Protocol buffer version of WALKey; see WALKey comment, not really a key but WALEdit header for some KVs
          + public void preWALRestore(ObserverContext<? extends RegionCoprocessorEnvironment> env, HRegionInfo info,
          + public boolean preWALWrite(ObserverContext<? extends WALCoprocessorEnvironment> ctx, HRegionInfo info,
          + public void postWALWrite(ObserverContext<? extends WALCoprocessorEnvironment> ctx, HRegionInfo info,
          + Path logDir = new Path(rootdir, DefaultWALProvider.getWALDirectoryName(serverName.toString()));
          + Path logDir = new Path(this.rootdir, DefaultWALProvider.getWALDirectoryName(serverName.toString()));
          + return createHRegion(info, rootDir, tableDir, conf, hTableDescriptor, wal, initialize, ignoreWAL);

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12681773/HBASE-10378.3.patch.txt against trunk revision . ATTACHMENT ID: 12681773 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 349 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 25 warning messages. -1 checkstyle . The applied patch generated 3861 checkstyle errors (more than the trunk's current 3788 errors). -1 findbugs . The patch appears to introduce 4 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: +set corecommands=shell master regionserver thrift thrift2 rest avro hlog wal hbck hfile zookeeper zkcli upgrade mapredcp + * Protocol buffer version of WALKey; see WALKey comment, not really a key but WALEdit header for some KVs + * Protocol buffer version of WALKey; see WALKey comment, not really a key but WALEdit header for some KVs +// Protocol buffer version of WALKey; see WALKey comment, not really a key but WALEdit header for some KVs + public void preWALRestore(ObserverContext<? extends RegionCoprocessorEnvironment> env, HRegionInfo info, + public boolean preWALWrite(ObserverContext<? extends WALCoprocessorEnvironment> ctx, HRegionInfo info, + public void postWALWrite(ObserverContext<? extends WALCoprocessorEnvironment> ctx, HRegionInfo info, + Path logDir = new Path(rootdir, DefaultWALProvider.getWALDirectoryName(serverName.toString())); + Path logDir = new Path(this.rootdir, DefaultWALProvider.getWALDirectoryName(serverName.toString())); + return createHRegion(info, rootDir, tableDir, conf, hTableDescriptor, wal, initialize, ignoreWAL); -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11692//console This message is automatically generated.
          Hide
          Sean Busbey added a comment -

          Latest patch out of reviewboard rebased onto current master.

          Show
          Sean Busbey added a comment - Latest patch out of reviewboard rebased onto current master.
          Hide
          Sean Busbey added a comment -

          Updating the release notes based on RB feedback and the current (yet to be posted) reworking of the code to maintain more backwards compatibility.

          I believe at this point there aren't any backwards incompatibilities with Phoenix's non-test code. There is impact in both unit and integration tests.

          The replication code is further into the guts of the write-ahead-log than the coprocessors were, so it has changed more. Projects that rely on hooking into that code path will be broken. This includes the only prominent example I could think of, the HBase Lily Indexer. Since they have a compatibility layer already, these changes should just mean that they need an updated one based on their current 0.98 shim that accounts for the class renames in this patch.

          Open to suggestions on other downstream impacts I should look for.

          Show
          Sean Busbey added a comment - Updating the release notes based on RB feedback and the current (yet to be posted) reworking of the code to maintain more backwards compatibility. I believe at this point there aren't any backwards incompatibilities with Phoenix's non-test code. There is impact in both unit and integration tests. The replication code is further into the guts of the write-ahead-log than the coprocessors were, so it has changed more. Projects that rely on hooking into that code path will be broken. This includes the only prominent example I could think of, the HBase Lily Indexer. Since they have a compatibility layer already, these changes should just mean that they need an updated one based on their current 0.98 shim that accounts for the class renames in this patch. Open to suggestions on other downstream impacts I should look for.
          Hide
          Sean Busbey added a comment -

          cancelled patch while I fix stack's last round of comments and I make the change to HLogKey binary compatible.

          Show
          Sean Busbey added a comment - cancelled patch while I fix stack's last round of comments and I make the change to HLogKey binary compatible.
          Hide
          Sean Busbey added a comment -

          Adding release notes in progress.

          Show
          Sean Busbey added a comment - Adding release notes in progress.
          Hide
          Sean Busbey added a comment -

          patch up on review board that keeps enough compatibility that I think it would work for branch-1. (ping Enis Soztutar)

          The current set of changes to the API should be sufficient to allow for multiple wals per region server, so I subsumed HBASE-8610 and attached its last targeted version.

          Show
          Sean Busbey added a comment - patch up on review board that keeps enough compatibility that I think it would work for branch-1. (ping Enis Soztutar ) The current set of changes to the API should be sufficient to allow for multiple wals per region server, so I subsumed HBASE-8610 and attached its last targeted version.
          Hide
          Sean Busbey added a comment -

          As a sanity check for the approach taken in this ticket I've outlined how I think I'll be able to get from here to the improvements for wals throughput, mttr, and write latency. If folks have any feedback on how this looks I'd appreciate it. Hopefully commenting on unrendered markdown in ReviewBoard isn't too terrible.

          Show
          Sean Busbey added a comment - As a sanity check for the approach taken in this ticket I've outlined how I think I'll be able to get from here to the improvements for wals throughput, mttr, and write latency . If folks have any feedback on how this looks I'd appreciate it. Hopefully commenting on unrendered markdown in ReviewBoard isn't too terrible.
          Hide
          Sean Busbey added a comment -

          This has turned into a bit of a time sink. Now that I have small tests passing, I'm posting some WIP to get suggestions early about parts that are worth breaking off into different tickets. Note that the RB patch is definitely not something that's ready to go in.

          RB description has the major compatibility goals and current failings. The github WIP commit message has questions that I'm trying to work out in my head. I'd love to get feedback on the compatibility stuff and suggestions on the open questions.

          Show
          Sean Busbey added a comment - This has turned into a bit of a time sink. Now that I have small tests passing, I'm posting some WIP to get suggestions early about parts that are worth breaking off into different tickets. Note that the RB patch is definitely not something that's ready to go in. github branch - has both the original work from this ticket rebased onto master and then follow on changes. reviewboard for the net changes on top of master (warning: it's 10 pages) RB description has the major compatibility goals and current failings. The github WIP commit message has questions that I'm trying to work out in my head. I'd love to get feedback on the compatibility stuff and suggestions on the open questions.
          Hide
          Himanshu Vashishtha added a comment -

          Thanks for the comments Stack. I would get back on this once I am back.

          Show
          Himanshu Vashishtha added a comment - Thanks for the comments Stack. I would get back on this once I am back.
          Hide
          stack added a comment -

          I like the doing away with the awful HLog name replacing it w/ WAL.

          I suppose we should remove NamespaceUpgrade.java now we are past out 0.96. But that is not for this issue.

          This is a bit odd, returning a 'Service' when we asked for a 'Log':

          • public HLog getLog() {
            + public WALService getLog() { return this.log; }

          Similar here, final WALService wal, in that the parameter name should be walService, not just wal... and here 'WALService log'

          Here in the comment it says:

          + * need to explicitly close the

          {@link WALService} it created too;

          ... but I'd think we'd start and stop a Service, not 'close' it. What you think?

          It shouldn't be '{@link WALService}

          file. ' a file here... how WALService is implemented should not be a concern of the user (whether it uses files or not). This is a 'nit'.

          Yeah, you 'stop' a service I'd say: '+ * The

          {@link WALService}

          for the created region needs to be closed explicitly.'

          Fix this:

          final WALService hlog,

          This should be cleaned up:

          + WALService effectiveHLog = hlog;

          something like '+ WALService effectiveWALService = walService;'

          Fix ' final WALService hlog)'

          + protected volatile WALService hlogForMeta; should be 'walServiceForMeta'.

          + private WALService getMetaWAL() throws IOException { should be getMetaWALService

          + protected WALService instantiateHLog( should be instantiateWALService...

          I think you get the point.

          I skipped forward to the Interfaces:

          It is a pity that rollWriter has to be let outside of WALService but there is probably little to do about it.

          stop rather than 'close'. Does it need a 'start'?

          + // TODO: Do we need all these versions of sync?
          + void hsync() throws IOException;
          +
          + void hflush() throws IOException;
          +
          + void sync() throws IOException;
          +
          + void sync(long txid) throws IOException;

          ?

          Does this have to be in the WALService – + WALCoprocessorHost getCoprocessorHost();?

          Otherwise, the WALService looks nice and clean.

          These belong in WALService instead?

          + /**
          + * @return the total number of WAL files (including rolled WALs).
          + */
          + int getNumLogFiles();
          +
          + /**
          + * returns the number of rolled WAL files.
          + */
          + int getNumRolledLogFiles();
          +
          + /**
          + * @return the path of the current WAL file.
          + */
          + Path getCurrentFileName();
          +

          + /**
          + * Get LowReplication-Roller status
          + * @return lowReplicationRollEnabled
          + */
          + boolean isLowReplicationRollEnabled();

          We should get rid of this:

          + // TODO: Remove this Writable.
          + class Entry implements Writable {

          Should just be the pb. But another patch.

          Otherwise WAL looks good.

          Show
          stack added a comment - I like the doing away with the awful HLog name replacing it w/ WAL. I suppose we should remove NamespaceUpgrade.java now we are past out 0.96. But that is not for this issue. This is a bit odd, returning a 'Service' when we asked for a 'Log': public HLog getLog() { + public WALService getLog() { return this.log; } Similar here, final WALService wal, in that the parameter name should be walService, not just wal... and here 'WALService log' Here in the comment it says: + * need to explicitly close the {@link WALService} it created too; ... but I'd think we'd start and stop a Service, not 'close' it. What you think? It shouldn't be '{@link WALService} file. ' a file here... how WALService is implemented should not be a concern of the user (whether it uses files or not). This is a 'nit'. Yeah, you 'stop' a service I'd say: '+ * The {@link WALService} for the created region needs to be closed explicitly.' Fix this: final WALService hlog, This should be cleaned up: + WALService effectiveHLog = hlog; something like '+ WALService effectiveWALService = walService;' Fix ' final WALService hlog)' + protected volatile WALService hlogForMeta; should be 'walServiceForMeta'. + private WALService getMetaWAL() throws IOException { should be getMetaWALService + protected WALService instantiateHLog( should be instantiateWALService... I think you get the point. I skipped forward to the Interfaces: It is a pity that rollWriter has to be let outside of WALService but there is probably little to do about it. stop rather than 'close'. Does it need a 'start'? + // TODO: Do we need all these versions of sync? + void hsync() throws IOException; + + void hflush() throws IOException; + + void sync() throws IOException; + + void sync(long txid) throws IOException; ? Does this have to be in the WALService – + WALCoprocessorHost getCoprocessorHost();? Otherwise, the WALService looks nice and clean. These belong in WALService instead? + /** + * @return the total number of WAL files (including rolled WALs). + */ + int getNumLogFiles(); + + /** + * returns the number of rolled WAL files. + */ + int getNumRolledLogFiles(); + + /** + * @return the path of the current WAL file. + */ + Path getCurrentFileName(); + + /** + * Get LowReplication-Roller status + * @return lowReplicationRollEnabled + */ + boolean isLowReplicationRollEnabled(); We should get rid of this: + // TODO: Remove this Writable. + class Entry implements Writable { Should just be the pb. But another patch. Otherwise WAL looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12624933/10378-2.patch
          against trunk revision .
          ATTACHMENT ID: 12624933

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 128 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12624933/10378-2.patch against trunk revision . ATTACHMENT ID: 12624933 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 128 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8507//console This message is automatically generated.
          Hide
          Himanshu Vashishtha added a comment -

          Fixed the javadoc warning. Please review. Thanks.

          Show
          Himanshu Vashishtha added a comment - Fixed the javadoc warning. Please review. Thanks.
          Hide
          Himanshu Vashishtha added a comment -

          Sounds great Ram krishna.
          Btw, I tried sending an email to ramakrishna.s.vasudevan@gmail.com, but it bounced back. Let me know your email id and we can chat more on it offline. Thanks.

          Show
          Himanshu Vashishtha added a comment - Sounds great Ram krishna. Btw, I tried sending an email to ramakrishna.s.vasudevan@gmail.com, but it bounced back. Let me know your email id and we can chat more on it offline. Thanks.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Since it requires user's input, we can also overload the API as per WAL requirement?

          Ok. Fine with it.

          Looks like we both are on same page re getWAL() api? That is, hiding the details in the impl.

          +1. We are on the same page.
          I can send you what I made on this WAL Grouping things.
          One issue would be with performance inside getWAL api. Currently getWAL is getting called on every region Opening in OPenRegionHandler. So when multiple regions of the same table arrive we would add up synchronization there and that would slow down reigon opening.

          Show
          ramkrishna.s.vasudevan added a comment - Since it requires user's input, we can also overload the API as per WAL requirement? Ok. Fine with it. Looks like we both are on same page re getWAL() api? That is, hiding the details in the impl. +1. We are on the same page. I can send you what I made on this WAL Grouping things. One issue would be with performance inside getWAL api. Currently getWAL is getting called on every region Opening in OPenRegionHandler. So when multiple regions of the same table arrive we would add up synchronization there and that would slow down reigon opening.
          Hide
          Himanshu Vashishtha added a comment -

          Yes, hiding the impl specific details from HRS is one big point of this feature. Looks like we both are on same page re getWAL() api? That is, hiding the details in the impl.

          Re: rollWriter()

          ... called from admin. Currently it returns set of regions that were flushed on calling rollWriter.

          No, it doesn't flush any region. It just returns regions which the caller should flush in order to archive the oldest WAL file. Actually, this return value is never used by the shell.
          Anyway, though rolling is an internal stuff, we have exposed it as an API in shell. I would again go the same route and let WAL impl decide how to deal with it. For example, TableGroupWALImpl could roll the logs for all tables. Since it requires user's input, we can also overload the API as per WAL requirement?
          As mentioned in the parent jira, I will take a stab at it once basic switching is done.

          Show
          Himanshu Vashishtha added a comment - Yes, hiding the impl specific details from HRS is one big point of this feature. Looks like we both are on same page re getWAL() api? That is, hiding the details in the impl. Re: rollWriter() ... called from admin. Currently it returns set of regions that were flushed on calling rollWriter. No, it doesn't flush any region. It just returns regions which the caller should flush in order to archive the oldest WAL file. Actually, this return value is never used by the shell. Anyway, though rolling is an internal stuff, we have exposed it as an API in shell. I would again go the same route and let WAL impl decide how to deal with it. For example, TableGroupWALImpl could roll the logs for all tables. Since it requires user's input, we can also overload the API as per WAL requirement? As mentioned in the parent jira, I will take a stab at it once basic switching is done.
          Hide
          ramkrishna.s.vasudevan added a comment - - edited

          Yes Stack. I meant the implementation of the WALService. May be we can have an Abstract implementation for those like where we call Sycner.start(), writer.start() etc.

          {edit}

          : Changed spelling

          Show
          ramkrishna.s.vasudevan added a comment - - edited Yes Stack. I meant the implementation of the WALService. May be we can have an Abstract implementation for those like where we call Sycner.start(), writer.start() etc. {edit} : Changed spelling
          Hide
          stack added a comment -

          should we have an implemenation for the WALservice where based on the number of hLogs those many syncer and writer threads...

          Or shouldn't the above be encapsulated inside the implementation and not pass out from behind the WALService Interface? (Which seems to be what you favor in your later comment ramkrishna.s.vasudevan)

          Show
          stack added a comment - should we have an implemenation for the WALservice where based on the number of hLogs those many syncer and writer threads... Or shouldn't the above be encapsulated inside the implementation and not pass out from behind the WALService Interface? (Which seems to be what you favor in your later comment ramkrishna.s.vasudevan )
          Hide
          ramkrishna.s.vasudevan added a comment -

          We could discuss offline too, not an issue. and then consolidate our thoughts here.
          >bq.Or, have the grouping logic in WAL impl and rewrite the getWAL to use this grouping knowledge? i.e., getWAL passes a Hregioninfo to the underlying Group-WAL-impl, and the underlying Group-wal impl returns a WALService instance based on its grouping.
          I would prefer this. Let HRS not deal with these things. The HRS at any point of time should only say instantiate WAL and based on what grouping or mechanism we select those many HLogs should be instantiated and the supporting feature like syncer,writers and replication for every HLog should be done inside the Group-WAL impl.
          Another API which I was not sure how to go about was the rollWriter() called from admin. Currently it returns set of regions that were flushed on calling rollWriter.
          Now it should be an array of regions or a list of regions. We can discuss offline, fine with it.

          Show
          ramkrishna.s.vasudevan added a comment - We could discuss offline too, not an issue. and then consolidate our thoughts here. >bq.Or, have the grouping logic in WAL impl and rewrite the getWAL to use this grouping knowledge? i.e., getWAL passes a Hregioninfo to the underlying Group-WAL-impl, and the underlying Group-wal impl returns a WALService instance based on its grouping. I would prefer this. Let HRS not deal with these things. The HRS at any point of time should only say instantiate WAL and based on what grouping or mechanism we select those many HLogs should be instantiated and the supporting feature like syncer,writers and replication for every HLog should be done inside the Group-WAL impl. Another API which I was not sure how to go about was the rollWriter() called from admin. Currently it returns set of regions that were flushed on calling rollWriter. Now it should be an array of regions or a list of regions. We can discuss offline, fine with it.
          Hide
          Himanshu Vashishtha added a comment -

          Thanks for taking a look at the patch. These are really good questions, and are applicable for the current WAL implementation as well.

          should we have an implemenation for the WALservice where based on the number of hLogs those many syncer and writer threads need to be started along with the replicaiton services for them? Currently the HRS just instantiates one HLog and starts them. What do you say?

          Yes, that is definitely one possible implementation.

          Re: getWAL() api wrt grouping WALs:
          Yes, the grouping logic needs to be extracted and made available, it can be either in HRS, or in the WAL implementor. One way is to use an array of WALService instances in HRegionServer, and let the WALGroup return an array to the region server? That way, the region server could invoke its usual method (append/sync, etc) just like it does currently? The grouping logic would be present in HRs in this case.
          Or, have the grouping logic in WAL impl and rewrite the getWAL to use this grouping knowledge? i.e., getWAL passes a Hregioninfo to the underlying Group-WAL-impl, and the underlying Group-wal impl returns a WALService instance based on its grouping. In either case, I see WALGroup can extend ÅbstractWAL interface. What do you think? Or, let's chat offline?

          Show
          Himanshu Vashishtha added a comment - Thanks for taking a look at the patch. These are really good questions, and are applicable for the current WAL implementation as well. should we have an implemenation for the WALservice where based on the number of hLogs those many syncer and writer threads need to be started along with the replicaiton services for them? Currently the HRS just instantiates one HLog and starts them. What do you say? Yes, that is definitely one possible implementation. Re: getWAL() api wrt grouping WALs: Yes, the grouping logic needs to be extracted and made available, it can be either in HRS, or in the WAL implementor. One way is to use an array of WALService instances in HRegionServer, and let the WALGroup return an array to the region server? That way, the region server could invoke its usual method (append/sync, etc) just like it does currently? The grouping logic would be present in HRs in this case. Or, have the grouping logic in WAL impl and rewrite the getWAL to use this grouping knowledge? i.e., getWAL passes a Hregioninfo to the underlying Group-WAL-impl, and the underlying Group-wal impl returns a WALService instance based on its grouping. In either case, I see WALGroup can extend ÅbstractWAL interface. What do you think? Or, let's chat offline?
          Hide
          ramkrishna.s.vasudevan added a comment -

          Himanshu Vashishtha
          Had a glance at the patch. The WAL and WALService looks good.
          I have some basic questions, incase of rollWriter and replications (including starting the syncer and writer threads), should we have an implemenation for the WALservice where based on the number of hLogs those many syncer and writer threads need to be started along with the replicaiton services for them?
          Currently the HRS just instantiates one HLog and starts them. What do you say?
          I was having an idea of defining an interface called WALGrouper and every region server would use an type of this WALGrouper and this grouper would know how many hLog instance he is creating and for every instance those syncer and writer threads would be started.
          And the api

          public WALService getWAL() {
          

          How will this make sense when there are more than one hlog for that RS? I know that in ur current implementation there are only 2 HLog and one among them is going to be active? But what if my multiwal impl is not that way and i may have more than one active HLog?
          I can discuss with you offline too on some my concerns and questions that I had while doing HBASE-8610.

          Show
          ramkrishna.s.vasudevan added a comment - Himanshu Vashishtha Had a glance at the patch. The WAL and WALService looks good. I have some basic questions, incase of rollWriter and replications (including starting the syncer and writer threads), should we have an implemenation for the WALservice where based on the number of hLogs those many syncer and writer threads need to be started along with the replicaiton services for them? Currently the HRS just instantiates one HLog and starts them. What do you say? I was having an idea of defining an interface called WALGrouper and every region server would use an type of this WALGrouper and this grouper would know how many hLog instance he is creating and for every instance those syncer and writer threads would be started. And the api public WALService getWAL() { How will this make sense when there are more than one hlog for that RS? I know that in ur current implementation there are only 2 HLog and one among them is going to be active? But what if my multiwal impl is not that way and i may have more than one active HLog? I can discuss with you offline too on some my concerns and questions that I had while doing HBASE-8610 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623840/10378-1.patch
          against trunk revision .
          ATTACHMENT ID: 12623840

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 128 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623840/10378-1.patch against trunk revision . ATTACHMENT ID: 12623840 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 128 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8469//console This message is automatically generated.
          Hide
          Himanshu Vashishtha added a comment -

          Attaching an initial patch implementing the above mentioned changes. Please have a look at the overall approach. Thanks.

          For testing, mvn clean test -PrunAllTests passes for both hadoop1 and hadoop2. Also ran the patched version on a test cluster/killing nodes and doing log splitting for basic sanity test. It looks good.

          Show
          Himanshu Vashishtha added a comment - Attaching an initial patch implementing the above mentioned changes. Please have a look at the overall approach. Thanks. For testing, mvn clean test -PrunAllTests passes for both hadoop1 and hadoop2. Also ran the patched version on a test cluster/killing nodes and doing log splitting for basic sanity test. It looks good.

            People

            • Assignee:
              Sean Busbey
              Reporter:
              Himanshu Vashishtha
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development