HBase
  1. HBase
  2. HBASE-6659

Port HBASE-6508 Filter out edits at log split time

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      0.96notable

      Description

      HBASE-6508 is for 0.89-fb branch.

      This JIRA ports the feature to trunk.

      1. 6508-v2.txt
        47 kB
        Ted Yu
      2. 6508-v3.txt
        47 kB
        Ted Yu
      3. 6508-v4.txt
        96 kB
        Ted Yu
      4. 6508-v5.txt
        94 kB
        Ted Yu
      5. 6508-v7.txt
        92 kB
        Ted Yu
      6. 6508-v8.txt
        96 kB
        Ted Yu
      7. 6508-v9.txt
        94 kB
        Ted Yu

        Activity

        Hide
        Ted Yu added a comment -

        Review request is published at https://reviews.apache.org/r/6768

        Show
        Ted Yu added a comment - Review request is published at https://reviews.apache.org/r/6768
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542369/6508.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2686//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542369/6508.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2686//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542382/6508-v2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 92 warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.TestCheckTestClasses
        org.apache.hadoop.hbase.io.TestHeapSize

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542382/6508-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 92 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestCheckTestClasses org.apache.hadoop.hbase.io.TestHeapSize Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2687//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542417/6508-v4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 92 warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.TestMultiVersions

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542417/6508-v4.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 92 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestMultiVersions Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2693//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        TestMultiVersions failure is not related to latest patch.

        Show
        Ted Yu added a comment - TestMultiVersions failure is not related to latest patch.
        Hide
        Ted Yu added a comment -

        Removed some white spaces.
        Plan to integrate this evening if there is no objection.

        Show
        Ted Yu added a comment - Removed some white spaces. Plan to integrate this evening if there is no objection.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542608/6508-v4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 92 warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542608/6508-v4.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 92 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542656/6508-v4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 92 warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542656/6508-v4.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 92 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Rebase for trunk.

        Show
        Ted Yu added a comment - Rebase for trunk.
        Hide
        Ted Yu added a comment -

        Integrated to trunk.

        Show
        Ted Yu added a comment - Integrated to trunk.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3291 (See https://builds.apache.org/job/HBase-TRUNK/3291/)
        HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1378631)

        Result = FAILURE
        Tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
        • /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
        • /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3291 (See https://builds.apache.org/job/HBase-TRUNK/3291/ ) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1378631) Result = FAILURE Tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
        Hide
        stack added a comment -

        Sorry. Meant to review this patch before it went in.

        I have a bit of problem w/ this patch in that it loads up new payloads on top of a channel we've been trying to shut down, the regionserver to master heartbeat.

        Here are some other comments:

        Are all the splitLogFile changes to add an extra null arg needed? Could do an overload for case where arg is non-null? (with default null)

        Do we not have the methods getRegionsByServer in HBaseTestingUtility already – or at least in the hbase cluster object?

        Could TestHLogFiltering tests not be added to the TestHLog? Seems related?

        We have to add this new regionserver call getLastFlushedSequenceId even though the regionservers are heartbeating every second (by default)?

        This seems like an ugly entanglement:

        + // For checking the latest flushed sequence id
        + protected final RegionServerStatusProtocol master;

        ... a master reference down deep inside in HLogSplitter. Its needed?

        We make a call to the master per region from down inside HLogSplitter to ask for last known sequence id?

        What is difference between:

        public FlushRegionResponse flushRegion(final RpcController controller,
        final FlushRegionRequest request) throws ServiceException {

        and

        + public void flushRegion(byte[] regionName)
        + throws NotServingRegionException, IOException {

        Can you not do what you want w/ the first method? Its pb'd? Not sure how you get at the second (You can pass null to the first IIRC... at least the second should call the first.

        Do we have to add

        + void updateLastFlushedSequenceIds(ServerName sn, ServerLoad hsl);

        to MasterServices? I'd like it if we could keep MasterServices curt. Everytime we add a new method, it makes for a bunch of changes in tests such as the bleow and it also makes mocking that little bit tougher.

        @Override
        + public void updateLastFlushedSequenceIds(ServerName sn, ServerLoad hsl)

        { + // no-op + }

        Do we have to add a new map like this:

        + private final SortedMap<byte[], Long> flushedSequenceIdByRegion =
        + new ConcurrentSkipListMap<byte[], Long>(Bytes.BYTES_COMPARATOR);

        Isn't there elsewhere where we keep list of regions out on cluster. Could we add the sequenceid as an attribute on that Map rather than introduce a new one?

        Putting sequenceid into serverload doesn't seem right. ServerLoad already is a dumping ground. Now we have pbs, could add as attribute easy enough.

        Show
        stack added a comment - Sorry. Meant to review this patch before it went in. I have a bit of problem w/ this patch in that it loads up new payloads on top of a channel we've been trying to shut down, the regionserver to master heartbeat. Here are some other comments: Are all the splitLogFile changes to add an extra null arg needed? Could do an overload for case where arg is non-null? (with default null) Do we not have the methods getRegionsByServer in HBaseTestingUtility already – or at least in the hbase cluster object? Could TestHLogFiltering tests not be added to the TestHLog? Seems related? We have to add this new regionserver call getLastFlushedSequenceId even though the regionservers are heartbeating every second (by default)? This seems like an ugly entanglement: + // For checking the latest flushed sequence id + protected final RegionServerStatusProtocol master; ... a master reference down deep inside in HLogSplitter. Its needed? We make a call to the master per region from down inside HLogSplitter to ask for last known sequence id? What is difference between: public FlushRegionResponse flushRegion(final RpcController controller, final FlushRegionRequest request) throws ServiceException { and + public void flushRegion(byte[] regionName) + throws NotServingRegionException, IOException { Can you not do what you want w/ the first method? Its pb'd? Not sure how you get at the second (You can pass null to the first IIRC... at least the second should call the first. Do we have to add + void updateLastFlushedSequenceIds(ServerName sn, ServerLoad hsl); to MasterServices? I'd like it if we could keep MasterServices curt. Everytime we add a new method, it makes for a bunch of changes in tests such as the bleow and it also makes mocking that little bit tougher. @Override + public void updateLastFlushedSequenceIds(ServerName sn, ServerLoad hsl) { + // no-op + } Do we have to add a new map like this: + private final SortedMap<byte[], Long> flushedSequenceIdByRegion = + new ConcurrentSkipListMap<byte[], Long>(Bytes.BYTES_COMPARATOR); Isn't there elsewhere where we keep list of regions out on cluster. Could we add the sequenceid as an attribute on that Map rather than introduce a new one? Putting sequenceid into serverload doesn't seem right. ServerLoad already is a dumping ground. Now we have pbs, could add as attribute easy enough.
        Hide
        Ted Yu added a comment -

        In hbase.proto, sequence Id is added to:

        message RegionLoad {
        

        ServerLoad.RegionLoad is just another form of representation for the above.
        In regionServerReport(), we have:

              HBaseProtos.ServerLoad sl = request.getLoad();
              this.serverManager.regionServerReport(ProtobufUtil.toServerName(request.getServer()), new ServerLoad(sl));
        

        If the regionserver to master heartbeat is not the place to channel sequence Id information, please remind me of the correct channel to use.

        I can revert the patch to facilitate further discussion.

        Show
        Ted Yu added a comment - In hbase.proto, sequence Id is added to: message RegionLoad { ServerLoad.RegionLoad is just another form of representation for the above. In regionServerReport(), we have: HBaseProtos.ServerLoad sl = request.getLoad(); this .serverManager.regionServerReport(ProtobufUtil.toServerName(request.getServer()), new ServerLoad(sl)); If the regionserver to master heartbeat is not the place to channel sequence Id information, please remind me of the correct channel to use. I can revert the patch to facilitate further discussion.
        Hide
        Ted Yu added a comment -

        w.r.t. updateLastFlushedSequenceIds(), since regionServerReport() is in ServerManager, we need a way to let master know region <-> sequence Id mapping.

        I have reverted the patch.

        Show
        Ted Yu added a comment - w.r.t. updateLastFlushedSequenceIds(), since regionServerReport() is in ServerManager, we need a way to let master know region <-> sequence Id mapping. I have reverted the patch.
        Hide
        stack added a comment -

        Again, sorry w/ my late review. Thanks for reverting the patch for now.

        Yeah, seems odd adding currentsequenceid to RegionLoad? Its not part of region load, I don't think. Its more a region attribute (this would not be the first parameter that cannot be properly called a 'load').

        High-level, we'd been trying to shrink the heartbeat with the hope that one day it'd be so small we could drown it in a bathtub. Maybe thats not possible. Outstanding is the fat serverload object updating the master on each heartbeat. I don't have specific issues to hand but we've talked about making it so this serverload could be obtained say via metrics/jmx since we are publishing the bulk of it there as well as over here up on the heartbeat; seems silly duplicating info. ServerLoad updating the master can be a bit lossy... same with this new sequenceid attribute. We could put it on the heartbeat – either as is where serverload is a dumping ground of all we want to pass from regionserver to master or we could put it in a data structure apart from serverload – and then maybe later do refactor of heartbeat to get its info elsewhere (from zk if unchanging post regionserver registration) and out of metrics w/ master pulling the info when its interested in particular?

        Show
        stack added a comment - Again, sorry w/ my late review. Thanks for reverting the patch for now. Yeah, seems odd adding currentsequenceid to RegionLoad? Its not part of region load, I don't think. Its more a region attribute (this would not be the first parameter that cannot be properly called a 'load'). High-level, we'd been trying to shrink the heartbeat with the hope that one day it'd be so small we could drown it in a bathtub. Maybe thats not possible. Outstanding is the fat serverload object updating the master on each heartbeat. I don't have specific issues to hand but we've talked about making it so this serverload could be obtained say via metrics/jmx since we are publishing the bulk of it there as well as over here up on the heartbeat; seems silly duplicating info. ServerLoad updating the master can be a bit lossy... same with this new sequenceid attribute. We could put it on the heartbeat – either as is where serverload is a dumping ground of all we want to pass from regionserver to master or we could put it in a data structure apart from serverload – and then maybe later do refactor of heartbeat to get its info elsewhere (from zk if unchanging post regionserver registration) and out of metrics w/ master pulling the info when its interested in particular?
        Hide
        Gregory Chanan added a comment -

        On Stack's comments:

        • I agree it's silly to duplicate stats. I even noticed once the stats weren't quite duplicated (I think the metrics calculates requestsPerSecond differently than ServerLoad does), which is pointless and confusing
        • On this specific issue: should we put this info into ZK? it's "transient state" in the sense that we can recover fine if it gets blown away, and it has the added advantage that if both the Master and RS go down (but ZK stays up), we can still run this optimization.
        Show
        Gregory Chanan added a comment - On Stack's comments: I agree it's silly to duplicate stats. I even noticed once the stats weren't quite duplicated (I think the metrics calculates requestsPerSecond differently than ServerLoad does), which is pointless and confusing On this specific issue: should we put this info into ZK? it's "transient state" in the sense that we can recover fine if it gets blown away, and it has the added advantage that if both the Master and RS go down (but ZK stays up), we can still run this optimization.
        Hide
        stack added a comment -

        I thought about putting it into zk. The good thing is that it'd be persisted so would make it across a master failover. In general the less footprint in zk, the better. If we went this route, it'd be a znode per region on the cluster. We'd write this region znode every time we flush. During high write times could flush a file per second or so.... If 100k regions on cluster... could be a bunch of zk writes.

        I suppose an rpc to the master to ask it about a long that its keeping in an in-memory map is way cheaper than going to fs, finding the region, and getting the sequenceid out of the hfile metadata.

        Show
        stack added a comment - I thought about putting it into zk. The good thing is that it'd be persisted so would make it across a master failover. In general the less footprint in zk, the better. If we went this route, it'd be a znode per region on the cluster. We'd write this region znode every time we flush. During high write times could flush a file per second or so.... If 100k regions on cluster... could be a bunch of zk writes. I suppose an rpc to the master to ask it about a long that its keeping in an in-memory map is way cheaper than going to fs, finding the region, and getting the sequenceid out of the hfile metadata.
        Hide
        Ted Yu added a comment -

        or we could put it in a data structure apart from serverload

        Do we have a route through which such data structure can be channeled to master nowadays ?

        out of metrics w/ master pulling the info when its interested in particular

        The use case for this feature is that log splitters may request sequence Ids in a short period of time. I am not sure if I understand the above comment enough in this context.

        I agree with Stack's analysis on using zookeeper. In a large cluster under heavy write load, sufficient experiment / profiling is needed to justify the approach.

        Again, the reason I came with the port was that I found this feature very helpful, even if HBASE-5699 is implemented.
        I think the patch didn't introduce fresh design deficiency.

        I can address Stack's other comments in the next patch.

        When there is good alternative to retiring heartbeat, all the payload can be moved in one shot.

        Show
        Ted Yu added a comment - or we could put it in a data structure apart from serverload Do we have a route through which such data structure can be channeled to master nowadays ? out of metrics w/ master pulling the info when its interested in particular The use case for this feature is that log splitters may request sequence Ids in a short period of time. I am not sure if I understand the above comment enough in this context. I agree with Stack's analysis on using zookeeper. In a large cluster under heavy write load, sufficient experiment / profiling is needed to justify the approach. Again, the reason I came with the port was that I found this feature very helpful, even if HBASE-5699 is implemented. I think the patch didn't introduce fresh design deficiency. I can address Stack's other comments in the next patch. When there is good alternative to retiring heartbeat, all the payload can be moved in one shot.
        Hide
        Gregory Chanan added a comment -

        Just so I'm sure I understand – lower footprint in ZK is better because ZK persists the data (absent ZOOKEEPER-866) or because of the overhead for consensus (I'm assuming the other costs, e.g. rpc time are equivalent in ZK vs HMaster, but maybe that's not correct)? Or both?

        Show
        Gregory Chanan added a comment - Just so I'm sure I understand – lower footprint in ZK is better because ZK persists the data (absent ZOOKEEPER-866 ) or because of the overhead for consensus (I'm assuming the other costs, e.g. rpc time are equivalent in ZK vs HMaster, but maybe that's not correct)? Or both?
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #154 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/154/)
        HBASE-6659 Port HBASE-6508 Filter out edits at log split time, revert (Revision 1378752)
        HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1378631)

        Result = FAILURE
        Tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
        • /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
        • /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java

        Tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
        • /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
        • /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #154 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/154/ ) HBASE-6659 Port HBASE-6508 Filter out edits at log split time, revert (Revision 1378752) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1378631) Result = FAILURE Tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java Tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
        Hide
        Ted Yu added a comment -

        @Stack:
        What should be the next step for this feature ?

        Show
        Ted Yu added a comment - @Stack: What should be the next step for this feature ?
        Hide
        stack added a comment -

        There are a few approaches I think. We could hold up this patch till we figure alternate channel for metrics and attributes – e.g. have the master listen to the metrics emitted by regionservers rather than have the regionserver blast load out on a metrics channel and duplicate their compostion putting same info out on the regionserver heartbeat – or we commit the patch as is (after the non-architectural issues raised in the issue have been addressed) and address the messaging problems later.

        It looks like it could help a bunch with MTTR even though its totally ugly the way we are calling master up out of an HLog (how do you put that facility behind an Interface?)

        Show
        stack added a comment - There are a few approaches I think. We could hold up this patch till we figure alternate channel for metrics and attributes – e.g. have the master listen to the metrics emitted by regionservers rather than have the regionserver blast load out on a metrics channel and duplicate their compostion putting same info out on the regionserver heartbeat – or we commit the patch as is (after the non-architectural issues raised in the issue have been addressed) and address the messaging problems later. It looks like it could help a bunch with MTTR even though its totally ugly the way we are calling master up out of an HLog (how do you put that facility behind an Interface?)
        Hide
        Ted Yu added a comment -

        or we commit the patch as is (after the non-architectural issues raised in the issue have been addressed) and address the messaging problems later.

        I would choose this route.

        Show
        Ted Yu added a comment - or we commit the patch as is (after the non-architectural issues raised in the issue have been addressed) and address the messaging problems later. I would choose this route.
        Hide
        Ted Yu added a comment -

        In HBaseTestingUtility, I don't see getRegionsByServer().
        TestHLog is a large test. There would be more tests added to TestHLog in the future. According to N's motto, we should keep runtime for large tests in check. So I keep TestHLogFiltering for now.
        New method getLastFlushedSequenceId() is called by HLogSplitter. The timing of this call is not in sync with region server heartbeat.

        We make a call to the master per region from down inside HLogSplitter to ask for last known sequence id?

        For 'protected final RegionServerStatusProtocol master', I agree there is something to be desired. Previously there was no need for HLogSplitter to know the actual status for any region in the WALs it splits. This feature adds this requirement. We can do buffering of calls to master. But I found that code would get more convoluted that way.

        w.r.t. introduction of void flushRegion(byte[] regionName), you're right that flushRegion(final RpcController controller, final FlushRegionRequest request) requires the composition of FlushRegionRequest. So that method is called by client.

        Will answer the remaining questions.

        Show
        Ted Yu added a comment - In HBaseTestingUtility, I don't see getRegionsByServer(). TestHLog is a large test. There would be more tests added to TestHLog in the future. According to N's motto, we should keep runtime for large tests in check. So I keep TestHLogFiltering for now. New method getLastFlushedSequenceId() is called by HLogSplitter. The timing of this call is not in sync with region server heartbeat. We make a call to the master per region from down inside HLogSplitter to ask for last known sequence id? For 'protected final RegionServerStatusProtocol master', I agree there is something to be desired. Previously there was no need for HLogSplitter to know the actual status for any region in the WALs it splits. This feature adds this requirement. We can do buffering of calls to master. But I found that code would get more convoluted that way. w.r.t. introduction of void flushRegion(byte[] regionName), you're right that flushRegion(final RpcController controller, final FlushRegionRequest request) requires the composition of FlushRegionRequest. So that method is called by client. Will answer the remaining questions.
        Hide
        Ted Yu added a comment -

        I mentioned the need for adding updateLastFlushedSequenceIds() @ 29/Aug/12 23:01
        Since 0.96 hasn't been released, I think adding methods to MasterServices should be acceptable.

        w.r.t. flushedSequenceIdByRegion map, there is no other map in HMaster which is keyed by byte[] (region name).

        Show
        Ted Yu added a comment - I mentioned the need for adding updateLastFlushedSequenceIds() @ 29/Aug/12 23:01 Since 0.96 hasn't been released, I think adding methods to MasterServices should be acceptable. w.r.t. flushedSequenceIdByRegion map, there is no other map in HMaster which is keyed by byte[] (region name).
        Hide
        Ted Yu added a comment -

        Patch v6 restores splitLogFile() method which doesn't take RegionServerStatusProtocol parameter.

        Show
        Ted Yu added a comment - Patch v6 restores splitLogFile() method which doesn't take RegionServerStatusProtocol parameter.
        Hide
        stack added a comment -

        Your responses are many and do not cite to what you are responding which makes it tough aligning what your response refers to. For example, this comment:

        In HBaseTestingUtility, I don't see getRegionsByServer().

        To what of my remarks is it in reply to?

        On the keeping TestHLogFiltering, thats fine. You give good rationale.

        The timing of this call is not in sync with region server heartbeat.

        Nevermind my comment; I had the workings mixed up.

        But I found that code would get more convoluted that way.

        Yes. Its super ugly rpc'ing the master from down deep in hlogsplitter. Could we put the rpc behind an Interface at least?

        w.r.t. introduction of void flushRegion(byte[] regionName), you're right that flushRegion(final RpcController controller, final FlushRegionRequest request) requires the composition of FlushRegionRequest. So that method is called by client.

        I don't follow what you are saying above.

        Since 0.96 hasn't been released, I think adding methods to MasterServices should be acceptable.

        My rationale for not adding methods has nothing to do w/ whether a release has been made or not; its about keeping these Interfaces as basic as possible so mocking does not become burdensome (when mocking is burdensome, fellas will skip on writing tests)

        w.r.t. flushedSequenceIdByRegion map, there is no other map in HMaster which is keyed by byte[] (region name).

        There is at least AM#regions

        Its use is different but I'd think some effort would be expended trying to make it so we don't proliferate state keepers per facility added to hbase. Maybe its not possible.

        Show
        stack added a comment - Your responses are many and do not cite to what you are responding which makes it tough aligning what your response refers to. For example, this comment: In HBaseTestingUtility, I don't see getRegionsByServer(). To what of my remarks is it in reply to? On the keeping TestHLogFiltering, thats fine. You give good rationale. The timing of this call is not in sync with region server heartbeat. Nevermind my comment; I had the workings mixed up. But I found that code would get more convoluted that way. Yes. Its super ugly rpc'ing the master from down deep in hlogsplitter. Could we put the rpc behind an Interface at least? w.r.t. introduction of void flushRegion(byte[] regionName), you're right that flushRegion(final RpcController controller, final FlushRegionRequest request) requires the composition of FlushRegionRequest. So that method is called by client. I don't follow what you are saying above. Since 0.96 hasn't been released, I think adding methods to MasterServices should be acceptable. My rationale for not adding methods has nothing to do w/ whether a release has been made or not; its about keeping these Interfaces as basic as possible so mocking does not become burdensome (when mocking is burdensome, fellas will skip on writing tests) w.r.t. flushedSequenceIdByRegion map, there is no other map in HMaster which is keyed by byte[] (region name). There is at least AM#regions Its use is different but I'd think some effort would be expended trying to make it so we don't proliferate state keepers per facility added to hbase. Maybe its not possible.
        Hide
        Ted Yu added a comment -

        Do we not have the methods getRegionsByServer in HBaseTestingUtility already

        'In HBaseTestingUtility, I don't see getRegionsByServer()' was for above comment.

        You can pass null to the first IIRC... at least the second should call the first.

        you're right that flushRegion(final RpcController controller, final FlushRegionRequest request) requires the composition of FlushRegionRequest.

        I will remove flushRegion(byte[] regionName) method and replace the call with one to the existing method.

        There is at least AM#regions

        I searched AssignmentManager and don't see regions field.

        w.r.t. adding updateLastFlushedSequenceIds(), the current formation of the patch is due to the fact that regionServerReport() is in ServerManager. ServerManager is marked @InterfaceAudience.Private, so ServerManager needs to deliver status update to HMaster.

        Could we put the rpc behind an Interface at least?

        To my understanding, RegionServerStatusProtocol is an interface:

        public interface RegionServerStatusProtocol extends
        

        Will upload new patch.

        Show
        Ted Yu added a comment - Do we not have the methods getRegionsByServer in HBaseTestingUtility already 'In HBaseTestingUtility, I don't see getRegionsByServer()' was for above comment. You can pass null to the first IIRC... at least the second should call the first. you're right that flushRegion(final RpcController controller, final FlushRegionRequest request) requires the composition of FlushRegionRequest. I will remove flushRegion(byte[] regionName) method and replace the call with one to the existing method. There is at least AM#regions I searched AssignmentManager and don't see regions field. w.r.t. adding updateLastFlushedSequenceIds(), the current formation of the patch is due to the fact that regionServerReport() is in ServerManager. ServerManager is marked @InterfaceAudience.Private, so ServerManager needs to deliver status update to HMaster. Could we put the rpc behind an Interface at least? To my understanding, RegionServerStatusProtocol is an interface: public interface RegionServerStatusProtocol extends Will upload new patch.
        Hide
        Ted Yu added a comment -

        Patch v7 removes flushRegion(byte[] regionName) and uses flushRegion(RpcController, FlushRegionRequest)

        TestHLogFiltering passes.

        Show
        Ted Yu added a comment - Patch v7 removes flushRegion(byte[] regionName) and uses flushRegion(RpcController, FlushRegionRequest) TestHLogFiltering passes.
        Hide
        Ted Yu added a comment -

        Previous Hadoop QA run was aborted.

        Show
        Ted Yu added a comment - Previous Hadoop QA run was aborted.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12543331/6508-v7.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 108 warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543331/6508-v7.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 108 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        TestReplication was missing in PreCommit build #2756.

        Other tests passed.

        I also updated https://reviews.apache.org/r/6768 with patch v7.

        Please provide your comments.

        Show
        Ted Yu added a comment - TestReplication was missing in PreCommit build #2756. Other tests passed. I also updated https://reviews.apache.org/r/6768 with patch v7. Please provide your comments.
        Hide
        stack added a comment -

        I searched AssignmentManager and don't see regions field.

        My bad. It was in 0.94. Its not in trunk (Jimmy removed it recently)

        w.r.t. adding updateLastFlushedSequenceIds(), the current formation of the patch is due to the fact that regionServerReport() is in ServerManager. ServerManager is marked @InterfaceAudience.Private, so ServerManager needs to deliver status update to HMaster.

        I do not follow. What is the above justifying?

        Should the map of sequenceids to regions be in ServerManager and not in Master so you don't have to leak it out to the HMaster wrapper class? (Would this mean you would not have to add updateLastFlushedSequenceIds to MasterServices? ServerManager deals w/ Load as it is... its tough because this stuff is so messy... load is carrying non-load attributes... keeping the mess confined to SM would be a bit of improvement over having to pass info from HMaster to SM and then back to HMaster)

        Could we put the rpc behind an Interface at least?

        What I meant is would it make sense making an Interface named LastSequenceId whose content was a method named:

        long getLastSequenceId(regionname);

        Its implementation would have RS invoke method on Master but at least we'd be making it so HLogSplitter didn't have to be polluted w/ some protocol implementation. Interface would also make it easier testing HLogSplitter.

        I took a look at v7. Most of my comments above are informed by a review of it up in rb

        Show
        stack added a comment - I searched AssignmentManager and don't see regions field. My bad. It was in 0.94. Its not in trunk (Jimmy removed it recently) w.r.t. adding updateLastFlushedSequenceIds(), the current formation of the patch is due to the fact that regionServerReport() is in ServerManager. ServerManager is marked @InterfaceAudience.Private, so ServerManager needs to deliver status update to HMaster. I do not follow. What is the above justifying? Should the map of sequenceids to regions be in ServerManager and not in Master so you don't have to leak it out to the HMaster wrapper class? (Would this mean you would not have to add updateLastFlushedSequenceIds to MasterServices? ServerManager deals w/ Load as it is... its tough because this stuff is so messy... load is carrying non-load attributes... keeping the mess confined to SM would be a bit of improvement over having to pass info from HMaster to SM and then back to HMaster) Could we put the rpc behind an Interface at least? What I meant is would it make sense making an Interface named LastSequenceId whose content was a method named: long getLastSequenceId(regionname); Its implementation would have RS invoke method on Master but at least we'd be making it so HLogSplitter didn't have to be polluted w/ some protocol implementation. Interface would also make it easier testing HLogSplitter. I took a look at v7. Most of my comments above are informed by a review of it up in rb
        Hide
        Ted Yu added a comment -

        load is carrying non-load attributes

        I will add javadoc as you suggested on review board.

        a bit of improvement over having to pass info from HMaster to SM and then back to HMaster

        Sequence Id information is passed from region servers to ServerManager, from ServerManager to Master, if that is what you meant.
        I checked a few methods in ServerManager, such as:

          public Map<ServerName, ServerLoad> getOnlineServers() {
          public Set<ServerName> getDeadServers() {
        

        They are all consumed in org.apache.hadoop.hbase.master namespace. Since ServerManager doesn't implement interface, currently I don't see how LastFlushedSequenceIds can be encapsulated within ServerManager.

        w.r.t. creating LastSequenceId interface, currently neither SplitLogWorker nor HLogSplitter has reference to RegionServerServices.
        If I understand your suggestion correctly, 'long getLastSequenceId(regionname)' can be added to RegionServerServices. We pass reference of RegionServerServices from region server to SplitLogWorker, then to HLogSplitter.
        HLogSplitter would be able to consume this new method.

        Once I get your confirmation, I would come up with patch v8.

        Show
        Ted Yu added a comment - load is carrying non-load attributes I will add javadoc as you suggested on review board. a bit of improvement over having to pass info from HMaster to SM and then back to HMaster Sequence Id information is passed from region servers to ServerManager, from ServerManager to Master, if that is what you meant. I checked a few methods in ServerManager, such as: public Map<ServerName, ServerLoad> getOnlineServers() { public Set<ServerName> getDeadServers() { They are all consumed in org.apache.hadoop.hbase.master namespace. Since ServerManager doesn't implement interface, currently I don't see how LastFlushedSequenceIds can be encapsulated within ServerManager. w.r.t. creating LastSequenceId interface, currently neither SplitLogWorker nor HLogSplitter has reference to RegionServerServices. If I understand your suggestion correctly, 'long getLastSequenceId(regionname)' can be added to RegionServerServices. We pass reference of RegionServerServices from region server to SplitLogWorker, then to HLogSplitter. HLogSplitter would be able to consume this new method. Once I get your confirmation, I would come up with patch v8.
        Hide
        Ted Yu added a comment -

        @Stack:
        What do you think of my proposal above ?

        Thanks

        Show
        Ted Yu added a comment - @Stack: What do you think of my proposal above ? Thanks
        Hide
        Ted Yu added a comment -

        For last flushed sequence Id, another option is to embed it in HRegionInfo.
        This way, there is no need to modify RegionLoad.

        Show
        Ted Yu added a comment - For last flushed sequence Id, another option is to embed it in HRegionInfo. This way, there is no need to modify RegionLoad.
        Hide
        stack added a comment -

        They are all consumed in org.apache.hadoop.hbase.master namespace. Since ServerManager doesn't implement interface, currently I don't see how LastFlushedSequenceIds can be encapsulated within ServerManager.

        Sorry. I don't follow what you are saying above.

        updateLastFlushedSequenceIds is implemented in HMaster and used by ServerManager. So ServerManager can use updateLastFlushedSequenceIds you must add it to MasterServices. Can you better encapsulate this new sequenceid to region accounting by moving it all into ServerManager so you do not have to do this back and forth between SM and HM and you can avoid adding more facility to the MasterServices Interface? Looking at updateLastFlushedSequenceIds implementation I do not see an obvious reason why we could not but I may be missing something.

        We pass reference of RegionServerServices from region server to SplitLogWorker, then to HLogSplitter. HLogSplitter would be able to consume this new method.

        The above is basically right (We'd pass a reference to a LastSequenceId Interface – though the actual instance may be a RegionServerServices that implements LastSequenceId).

        Do you think this makes sense? Would it be a PITA to do? We'd pass this Interface instead of passing in the crazy RegionServerStatusProtocol masterRef (A LastSequenceId would be easy to mock, a RegionServerStatusProtocol would be a pain)

        Show
        stack added a comment - They are all consumed in org.apache.hadoop.hbase.master namespace. Since ServerManager doesn't implement interface, currently I don't see how LastFlushedSequenceIds can be encapsulated within ServerManager. Sorry. I don't follow what you are saying above. updateLastFlushedSequenceIds is implemented in HMaster and used by ServerManager. So ServerManager can use updateLastFlushedSequenceIds you must add it to MasterServices. Can you better encapsulate this new sequenceid to region accounting by moving it all into ServerManager so you do not have to do this back and forth between SM and HM and you can avoid adding more facility to the MasterServices Interface? Looking at updateLastFlushedSequenceIds implementation I do not see an obvious reason why we could not but I may be missing something. We pass reference of RegionServerServices from region server to SplitLogWorker, then to HLogSplitter. HLogSplitter would be able to consume this new method. The above is basically right (We'd pass a reference to a LastSequenceId Interface – though the actual instance may be a RegionServerServices that implements LastSequenceId). Do you think this makes sense? Would it be a PITA to do? We'd pass this Interface instead of passing in the crazy RegionServerStatusProtocol masterRef (A LastSequenceId would be easy to mock, a RegionServerStatusProtocol would be a pain)
        Hide
        Ted Yu added a comment -

        Patch v8 is also on review board.

        Hopefully I have addressed Stack's comments.

        Show
        Ted Yu added a comment - Patch v8 is also on review board. Hopefully I have addressed Stack's comments.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544072/6508-v8.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 107 warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544072/6508-v8.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 107 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//console This message is automatically generated.
        Hide
        stack added a comment -

        +1

        To me at least, we encapsulate the sequenceid counting inside ServerManager rather than have it spread between SM and HM (also, don't need to add an esoteric method to the generic MasterServices Interface). HLogSplitter is passed a specific Interface for getting last sequenceid rather than a protocol implementation. I think this better. Thanks for entertaining my reviews.

        On commit remove '+ * Copyright The Apache Software Foundation' and change this '+ * @param master' (param name is different).

        This change seems gratuitous but it is from original patch so its ok:

        -        regionInfo.getTableName(), completeSequenceId,
        +        regionInfo.getTableName(), completeSeqId,
        

        Are the imports in MasterServices needed? (they don't seem to be referred to by this patch). Remove on commit.

        Show
        stack added a comment - +1 To me at least, we encapsulate the sequenceid counting inside ServerManager rather than have it spread between SM and HM (also, don't need to add an esoteric method to the generic MasterServices Interface). HLogSplitter is passed a specific Interface for getting last sequenceid rather than a protocol implementation. I think this better. Thanks for entertaining my reviews. On commit remove '+ * Copyright The Apache Software Foundation' and change this '+ * @param master' (param name is different). This change seems gratuitous but it is from original patch so its ok: - regionInfo.getTableName(), completeSequenceId, + regionInfo.getTableName(), completeSeqId, Are the imports in MasterServices needed? (they don't seem to be referred to by this patch). Remove on commit.
        Hide
        stack added a comment -

        Maybe check out the TestHLogMethods failure... unlikely it this change but worth a look I'd say.

        Show
        stack added a comment - Maybe check out the TestHLogMethods failure... unlikely it this change but worth a look I'd say.
        Hide
        Ted Yu added a comment -

        Patch v9 should be close to the final version - fixed compilation error.

        Show
        Ted Yu added a comment - Patch v9 should be close to the final version - fixed compilation error.
        Hide
        stack added a comment -

        I took a quick look. +1.

        Show
        stack added a comment - I took a quick look. +1.
        Hide
        ramkrishna.s.vasudevan added a comment -

        @Ted
        Nice useful backport. Thanks for taking it up.

        Show
        ramkrishna.s.vasudevan added a comment - @Ted Nice useful backport. Thanks for taking it up.
        Hide
        Ted Yu added a comment -
        Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.17 sec
        Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
        Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.567 sec
        Running org.apache.hadoop.hbase.regionserver.wal.TestHLogFiltering
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.173 sec
        Running org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.719 sec
        Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
        Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 234.204 sec
        
        Results :
        
        Tests run: 47, Failures: 0, Errors: 0, Skipped: 1
        

        Going to commit.

        Show
        Ted Yu added a comment - Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.17 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLog Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.567 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogFiltering Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.173 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.719 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 234.204 sec Results : Tests run: 47, Failures: 0, Errors: 0, Skipped: 1 Going to commit.
        Hide
        Ted Yu added a comment -

        Integrated to trunk.

        Thanks for the review, Stack and Ram.

        Show
        Ted Yu added a comment - Integrated to trunk. Thanks for the review, Stack and Ram.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3310 (See https://builds.apache.org/job/HBase-TRUNK/3310/)
        HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
        • /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
        • /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3310 (See https://builds.apache.org/job/HBase-TRUNK/3310/ ) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #163 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/163/)
        HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
        • /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
        • /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #163 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/163/ ) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
        Hide
        stack added a comment -

        Marking closed.

        Show
        stack added a comment - Marking closed.

          People

          • Assignee:
            Ted Yu
            Reporter:
            Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development