HBase
  1. HBase
  2. HBASE-10018

Remove region location prefetching

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.0, 0.96.0
    • Fix Version/s: 0.99.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The location prefetch features is removed from the code. The related interfaces are kept for backward compatibility, but do nothing.

      Description

      Issues with prefetching are:

      • we do two calls to meta: one for the exact row, one for the prefetch
      • it's done in a lock
      • we take the next 10 regions. Why 10, why the 10 next?
      • is it useful if the table has 100K regions?

      Options are:

      • just remove it
      • replace it with a reverse scan: this would save a call
      1. 10018v3.patch
        27 kB
        Nicolas Liochon
      2. 10018.v6.patch
        36 kB
        Nicolas Liochon
      3. 10018.v5.patch
        36 kB
        Nicolas Liochon
      4. 10018.v4.patch
        46 kB
        Nicolas Liochon
      5. 10018.v2.patch
        27 kB
        Nicolas Liochon
      6. 10018.v1.patch
        22 kB
        Nicolas Liochon

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          ABORTED: Integrated in HBase-TRUNK #5070 (See https://builds.apache.org/job/HBase-TRUNK/5070/)
          HBASE-10018 Change the location prefetch (nkeywal: rev 1585518)

          • /hbase/trunk/bin/region_mover.rb
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionKey.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
          • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java
          • /hbase/trunk/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
          Show
          Hudson added a comment - ABORTED: Integrated in HBase-TRUNK #5070 (See https://builds.apache.org/job/HBase-TRUNK/5070/ ) HBASE-10018 Change the location prefetch (nkeywal: rev 1585518) /hbase/trunk/bin/region_mover.rb /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionKey.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java /hbase/trunk/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
          Hide
          Nicolas Liochon added a comment -

          I'm not sure for HBASE-6841: for the last comment from Amit, it could be that the bottleneck is in the meta server, the lock could be just a red herring? On this other hand, if disabling the prefect worked for you, then yes HBASE-6841 is solved.

          Show
          Nicolas Liochon added a comment - I'm not sure for HBASE-6841 : for the last comment from Amit, it could be that the bottleneck is in the meta server, the lock could be just a red herring? On this other hand, if disabling the prefect worked for you, then yes HBASE-6841 is solved.
          Hide
          Nicolas Liochon added a comment -

          It's not too late to change the title at least. Let me do it.

          Show
          Nicolas Liochon added a comment - It's not too late to change the title at least. Let me do it.
          Hide
          Jean-Daniel Cryans added a comment -

          I guess I'm 5 minutes too late, but in the future would it be possible to set a better jira title once we know what the patch is going to look like? In this case, "location prefetch" isn't a thing, it's "meta prefetching" or "region location prefetching". Also it wasn't changed, it was removed.

          I think we can now close this issue? HBASE-6841

          Show
          Jean-Daniel Cryans added a comment - I guess I'm 5 minutes too late, but in the future would it be possible to set a better jira title once we know what the patch is going to look like? In this case, "location prefetch" isn't a thing, it's "meta prefetching" or "region location prefetching". Also it wasn't changed, it was removed. I think we can now close this issue? HBASE-6841
          Hide
          Nicolas Liochon added a comment -

          Committed, thanks for the reviews, Stack & Enis.

          Show
          Nicolas Liochon added a comment - Committed, thanks for the reviews, Stack & Enis.
          Hide
          Nicolas Liochon added a comment - - edited

          This looks like test flakiness. Commit is under way.

          Show
          Nicolas Liochon added a comment - - edited This looks like test flakiness. Commit is under way.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12638985/10018.v6.patch
          against trunk revision .
          ATTACHMENT ID: 12638985

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.procedure.TestZKProcedure

          -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.master.TestMasterNoCluster.testNotPullingDeadRegionServerFromZK(TestMasterNoCluster.java:298)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12638985/10018.v6.patch against trunk revision . ATTACHMENT ID: 12638985 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.procedure.TestZKProcedure -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.master.TestMasterNoCluster.testNotPullingDeadRegionServerFromZK(TestMasterNoCluster.java:298) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9214//console This message is automatically generated.
          Hide
          Enis Soztutar added a comment -

          +1

          Show
          Enis Soztutar added a comment - +1
          Hide
          Nicolas Liochon added a comment -

          v6 fixes TestClientTimeouts
          TestVisibilityLabelsWithDistributedLogReplay works here, it's likely unrelated.

          Show
          Nicolas Liochon added a comment - v6 fixes TestClientTimeouts TestVisibilityLabelsWithDistributedLogReplay works here, it's likely unrelated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637794/10018.v5.patch
          against trunk revision .
          ATTACHMENT ID: 12637794

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestClientTimeouts
          org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637794/10018.v5.patch against trunk revision . ATTACHMENT ID: 12637794 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestClientTimeouts org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9138//console This message is automatically generated.
          Hide
          Nicolas Liochon added a comment -

          I will commit v5 is hadoop qa is happy and if there is no objection...

          Show
          Nicolas Liochon added a comment - I will commit v5 is hadoop qa is happy and if there is no objection...
          Hide
          Nicolas Liochon added a comment -

          v5 = v4 rebased + deprecation of getClosestRowOrBefore

          getClosestRowOrBefore is actually deprecated in HTableInterface since 0.92: "As of version 0.92 this method is deprecated without replacement."
          But we still use it in a few place, and we can't remove it w/o breaking rolling upgrades.

          Show
          Nicolas Liochon added a comment - v5 = v4 rebased + deprecation of getClosestRowOrBefore getClosestRowOrBefore is actually deprecated in HTableInterface since 0.92: "As of version 0.92 this method is deprecated without replacement." But we still use it in a few place, and we can't remove it w/o breaking rolling upgrades.
          Hide
          stack added a comment -

          Patch looks great (on skim). You don't deprecated getClosestRowOrBefore. It is needed? If not, deprecate it on commit.

          Show
          stack added a comment - Patch looks great (on skim). You don't deprecated getClosestRowOrBefore. It is needed? If not, deprecate it on commit.
          Hide
          stack added a comment -

          +1 on 1 for your stated reasons.

          Show
          stack added a comment - +1 on 1 for your stated reasons.
          Hide
          Nicolas Liochon added a comment -

          I would like this to make it to trunk soon. Any feedback? I had a few takers at the beginning

          Show
          Nicolas Liochon added a comment - I would like this to make it to trunk soon. Any feedback? I had a few takers at the beginning
          Hide
          Nicolas Liochon added a comment -

          Any opinion? By default I would go for the patch as it is.

          Show
          Nicolas Liochon added a comment - Any opinion? By default I would go for the patch as it is.
          Hide
          Nicolas Liochon added a comment -

          I've just committed HBASE-9999. This patch is ready to go.
          I see two options:
          1) As it is, i.e. full removal of the features, interfaces kept deprecated for backward compatibility but they do nothing
          2) Change the patch to keep the prefetch as an option, deactivated by default.

          I've got a small preference for 1), but I don't mind doing 2).
          The reason for 2) would be that if there is a performance degradation for some use cases, we can use the option to to keep us safe.
          My reasons for prefering 1) are:

          • I remove much more code this way: with 2, I'm unclear about what to do with the existing code to put/remove a table in the "to prefect" list.
          • It's often better to have an optimized code path rather than 2 average.
          • an option non activated will become more or less obsolete or buggy very quickly, so it may not help that much...

          As you like guys

          Show
          Nicolas Liochon added a comment - I've just committed HBASE-9999 . This patch is ready to go. I see two options: 1) As it is, i.e. full removal of the features, interfaces kept deprecated for backward compatibility but they do nothing 2) Change the patch to keep the prefetch as an option, deactivated by default. I've got a small preference for 1), but I don't mind doing 2). The reason for 2) would be that if there is a performance degradation for some use cases, we can use the option to to keep us safe. My reasons for prefering 1) are: I remove much more code this way: with 2, I'm unclear about what to do with the existing code to put/remove a table in the "to prefect" list. It's often better to have an optimized code path rather than 2 average. an option non activated will become more or less obsolete or buggy very quickly, so it may not help that much... As you like guys
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12632321/10018.v4.patch
          against trunk revision .
          ATTACHMENT ID: 12632321

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12632321/10018.v4.patch against trunk revision . ATTACHMENT ID: 12632321 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8868//console This message is automatically generated.
          Hide
          Nicolas Liochon added a comment -

          I've solved the issue by filtering on the system tables on the scanner in the test. It seems to be the right thing to do, but I'm not sure of what the test is actually doing.
          v4 includes HBASE-9999.

          Show
          Nicolas Liochon added a comment - I've solved the issue by filtering on the system tables on the scanner in the test. It seems to be the right thing to do, but I'm not sure of what the test is actually doing. v4 includes HBASE-9999 .
          Hide
          Nicolas Liochon added a comment -

          I reproduce the issue, it's a table not found on hbase:namespace when there is a master copro. Looking.

          Show
          Nicolas Liochon added a comment - I reproduce the issue, it's a table not found on hbase:namespace when there is a master copro. Looking.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12632205/10018v3.patch
          against trunk revision .
          ATTACHMENT ID: 12632205

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.util.TestCoprocessorScanPolicy

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12632205/10018v3.patch against trunk revision . ATTACHMENT ID: 12632205 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.util.TestCoprocessorScanPolicy Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8861//console This message is automatically generated.
          Hide
          Nicolas Liochon added a comment -

          v3 fixes the test issue.
          It seems that small scans optimization is not there for small scan. We need it to make this patch efficient.

          Show
          Nicolas Liochon added a comment - v3 fixes the test issue. It seems that small scans optimization is not there for small scan. We need it to make this patch efficient.
          Hide
          Nicolas Liochon added a comment -

          btw, I reproduce the test issue. Will fix.

          Show
          Nicolas Liochon added a comment - btw, I reproduce the test issue. Will fix.
          Hide
          Nicolas Liochon added a comment -

          How much we prefetch when we do the reverse scan?

          Just one. We could increase this, with a kind of heuristic to stay in 1 network packet, but well..

          Should we get rid of HTable.getRowOrBefore()

          It's still used when you do a meta scan, even it's may be an historical case. I will double check. In any case, we need to deprecate it first.

          but implement it via small reverse scan?

          We could do this, but I'm not sure that this feature is very useful outside of the .meta. case, so...

          this will bring a regression

          You mean a performance regression? It's complex, because we unlikely to be better in all cases, and there is the usual difference between throughput and latency. And may be the reversed scan is not efficient enough (I need to look at how it works when the scan is 'small', I'm not sure this case is optimized actually).

          Show
          Nicolas Liochon added a comment - How much we prefetch when we do the reverse scan? Just one. We could increase this, with a kind of heuristic to stay in 1 network packet, but well.. Should we get rid of HTable.getRowOrBefore() It's still used when you do a meta scan, even it's may be an historical case. I will double check. In any case, we need to deprecate it first. but implement it via small reverse scan? We could do this, but I'm not sure that this feature is very useful outside of the .meta. case, so... this will bring a regression You mean a performance regression? It's complex, because we unlikely to be better in all cases, and there is the usual difference between throughput and latency. And may be the reversed scan is not efficient enough (I need to look at how it works when the scan is 'small', I'm not sure this case is optimized actually).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12631309/10018.v2.patch
          against trunk revision .
          ATTACHMENT ID: 12631309

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestClientNoCluster

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12631309/10018.v2.patch against trunk revision . ATTACHMENT ID: 12631309 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestClientNoCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8823//console This message is automatically generated.
          Hide
          Enis Soztutar added a comment -

          Before removing the prefetch, I think we should have an idea whether this will bring a regression. Hopefully not, but would be good if we can have a better idea.

          replace it with a reverse scan: this would save a call

          +1 to that. Makes sense to dog food. Should we get rid of HTable.getRowOrBefore() or keep that as an interface, but implement it via small reverse scan?
          This is good, even though we might still have prefetch.

          +    @Deprecated
               public void setRegionCachePrefetch(final byte[] tableName,
                   final boolean enable) {
          
          Show
          Enis Soztutar added a comment - Before removing the prefetch, I think we should have an idea whether this will bring a regression. Hopefully not, but would be good if we can have a better idea. replace it with a reverse scan: this would save a call +1 to that. Makes sense to dog food. Should we get rid of HTable.getRowOrBefore() or keep that as an interface, but implement it via small reverse scan? This is good, even though we might still have prefetch. + @Deprecated public void setRegionCachePrefetch( final byte [] tableName, final boolean enable) {
          Hide
          stack added a comment -

          Skimmed. Looks great. How much we prefetch when we do the reverse scan?

          Show
          stack added a comment - Skimmed. Looks great. How much we prefetch when we do the reverse scan?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12631242/10018.v1.patch
          against trunk revision .
          ATTACHMENT ID: 12631242

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.util.TestCoprocessorScanPolicy

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12631242/10018.v1.patch against trunk revision . ATTACHMENT ID: 12631242 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.util.TestCoprocessorScanPolicy Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8819//console This message is automatically generated.
          Hide
          Nicolas Liochon added a comment -

          v2: creates directly the scanner instead of going through a HTable creation, saving some subjects & configuration parsing.
          However, I seems that the Reversed scan doesn't manage the 'small' status. If it's the case, it will worth fixing here.

          Show
          Nicolas Liochon added a comment - v2: creates directly the scanner instead of going through a HTable creation, saving some subjects & configuration parsing. However, I seems that the Reversed scan doesn't manage the 'small' status. If it's the case, it will worth fixing here.
          Hide
          Nicolas Liochon added a comment -

          v1:

          • remove the lock
          • does a small reversed scan instead of a getRowSomething and a scan
          • deprecates all the prefetch methods, keep the interfaces but the methods do nothing
          • when we were doing a locate without the cache, we were also removing the entry from the cache. We don't do that anymore.
          Show
          Nicolas Liochon added a comment - v1: remove the lock does a small reversed scan instead of a getRowSomething and a scan deprecates all the prefetch methods, keep the interfaces but the methods do nothing when we were doing a locate without the cache, we were also removing the entry from the cache. We don't do that anymore.
          Hide
          Andrew Purtell added a comment -

          Moving out of 0.98.0. Put back if ready before RC.

          Show
          Andrew Purtell added a comment - Moving out of 0.98.0. Put back if ready before RC.
          Hide
          Andrew Purtell added a comment -

          +1 remove in 0.98 and disable it in 0.96. Planning on branching 0.98 on 12/1.

          Show
          Andrew Purtell added a comment - +1 remove in 0.98 and disable it in 0.96. Planning on branching 0.98 on 12/1.
          Hide
          stack added a comment -

          . If nobody disagrees, I will do that: "make it optional, deactivated by default, on .96 and .98."

          Fine by me. Since you are of a jacobin bent, you could remove in 0.98 and just configure it off in 0.96?

          Show
          stack added a comment - . If nobody disagrees, I will do that: "make it optional, deactivated by default, on .96 and .98." Fine by me. Since you are of a jacobin bent, you could remove in 0.98 and just configure it off in 0.96?
          Hide
          Nicolas Liochon added a comment -

          Does the prefetch have to be done inline

          Not really. It's just a little bit more complicated if we start a thread.

          What to do though if 1M regions

          It does not care: it reads the next 10 regions, whatever the table size.
          Ihmo, when you have a lot of region it does not really work, when you have just a few it's not really necessary.
          Excepted in the scenario mentioned by Elliott above.

          Could just flip the configuration to turn off prefetch in 0.98?

          I would prefer to be more radical: just remove it. Or at least, make it optional, deactivated by default, on .96 and .98.
          Thinking more about it, that's my prefered scenario. If nobody disagrees, I will do that: "make it optional, deactivated by default, on .96 and .98."

          Show
          Nicolas Liochon added a comment - Does the prefetch have to be done inline Not really. It's just a little bit more complicated if we start a thread. What to do though if 1M regions It does not care: it reads the next 10 regions, whatever the table size. Ihmo, when you have a lot of region it does not really work, when you have just a few it's not really necessary. Excepted in the scenario mentioned by Elliott above. Could just flip the configuration to turn off prefetch in 0.98? I would prefer to be more radical: just remove it. Or at least, make it optional, deactivated by default, on .96 and .98. Thinking more about it, that's my prefered scenario. If nobody disagrees, I will do that: "make it optional, deactivated by default, on .96 and .98."
          Hide
          stack added a comment -

          Does the prefetch have to be done inline? Can it be lazy? What to do though if 1M regions? What do we fetch? Maybe we don't want prefetch when we now have a bounded size for our cache. Could just flip the configuration to turn off prefetch in 0.98? When the prefetch was added, i thought there was also added a facility for reading in all of the meta – reading from HDFS IIRC? – but I can't find it so probably a pigment of my emancipation.

          Show
          stack added a comment - Does the prefetch have to be done inline? Can it be lazy? What to do though if 1M regions? What do we fetch? Maybe we don't want prefetch when we now have a bounded size for our cache. Could just flip the configuration to turn off prefetch in 0.98? When the prefetch was added, i thought there was also added a facility for reading in all of the meta – reading from HDFS IIRC? – but I can't find it so probably a pigment of my emancipation.
          Hide
          Elliott Clark added a comment -

          I would be all for removing it. The reverse scan wouldn't work for anyone that's scanning forward.

          Show
          Elliott Clark added a comment - I would be all for removing it. The reverse scan wouldn't work for anyone that's scanning forward.
          Hide
          Andrew Purtell added a comment -

          IIRC, a colleague at Trend Micro contributed this code to speed up HBase MR job initialization. There have been many changes to the client since, makes sense to redo this for current conditions.

          Show
          Andrew Purtell added a comment - IIRC, a colleague at Trend Micro contributed this code to speed up HBase MR job initialization. There have been many changes to the client since, makes sense to redo this for current conditions.

            People

            • Assignee:
              Nicolas Liochon
              Reporter:
              Nicolas Liochon
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development