Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 0.98.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede.

      I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not.

      1. HBASE-9857-trunk.patch
        63 kB
        Andrew Purtell
      2. HBASE-9857-trunk.patch
        66 kB
        Andrew Purtell
      3. HBASE-9857-0.98.patch
        64 kB
        Andrew Purtell
      4. 9857.patch
        61 kB
        Andrew Purtell
      5. 9857.patch
        61 kB
        Andrew Purtell

        Activity

        Hide
        Lars Hofhansl added a comment -

        How does this generally stack up against client triggered prefetching? I.e. the client would schedule the next partial scan ahead of time.

        Show
        Lars Hofhansl added a comment - How does this generally stack up against client triggered prefetching? I.e. the client would schedule the next partial scan ahead of time.
        Hide
        Andrew Purtell added a comment -

        The reads would not count as cache misses. There would be no RPCs. Only HFileScanner is involved, we are just loading blocks not looking at any keys.

        Show
        Andrew Purtell added a comment - The reads would not count as cache misses. There would be no RPCs. Only HFileScanner is involved, we are just loading blocks not looking at any keys.
        Hide
        Nick Dimiduk added a comment -

        This is a nice feature. I skimmed the patch, don't see why it's limited to HFileV3. Can it be made a general feature?

        I think it could be smart about loading the blocks, load either sequentially or over a random distribution until the cache is full. The "until full" part seems tricky as eviction detection isn't very straight-forward, or at least hasn't been so thus far in my work on HBASE-9806.

        Show
        Nick Dimiduk added a comment - This is a nice feature. I skimmed the patch, don't see why it's limited to HFileV3. Can it be made a general feature? I think it could be smart about loading the blocks, load either sequentially or over a random distribution until the cache is full. The "until full" part seems tricky as eviction detection isn't very straight-forward, or at least hasn't been so thus far in my work on HBASE-9806 .
        Hide
        Andrew Purtell added a comment -

        Thanks for looking at the patch Nick Dimiduk.

        don't see why it's limited to HFileV3. Can it be made a general feature

        I put the preload logic into the v3 reader because v3 is 'experimental'. Could trivially go into the v2 reader instead.

        I think it could be smart about loading the blocks, load either sequentially or over a random distribution until the cache is full

        Files to be preloaded are queued and scheduled to be handled by a small threadpool. When a thread picks up work for a file, the blocks are loaded sequentially using a non-pread scanner from offset 0 to the end of the index.

        By random did you mean randomly select work from the file queue?

        The "until full" part seems tricky as eviction detection isn't very straight-forward

        Right. If we had it, I could make use of it.

        Show
        Andrew Purtell added a comment - Thanks for looking at the patch Nick Dimiduk . don't see why it's limited to HFileV3. Can it be made a general feature I put the preload logic into the v3 reader because v3 is 'experimental'. Could trivially go into the v2 reader instead. I think it could be smart about loading the blocks, load either sequentially or over a random distribution until the cache is full Files to be preloaded are queued and scheduled to be handled by a small threadpool. When a thread picks up work for a file, the blocks are loaded sequentially using a non-pread scanner from offset 0 to the end of the index. By random did you mean randomly select work from the file queue? The "until full" part seems tricky as eviction detection isn't very straight-forward Right. If we had it, I could make use of it.
        Hide
        Andrew Purtell added a comment -

        Rebase on latest trunk

        Show
        Andrew Purtell added a comment - Rebase on latest trunk
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614106/9857.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 26 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614106/9857.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 26 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7887//console This message is automatically generated.
        Hide
        stack added a comment -

        This patch lgtm. Would need a rebase at this stage and fix of the javadoc warning above if this patch responsible but if you commit it, please add better doc around say the static key in HCD. This would be the location for authoritative doc on this feature.

        Show
        stack added a comment - This patch lgtm. Would need a rebase at this stage and fix of the javadoc warning above if this patch responsible but if you commit it, please add better doc around say the static key in HCD. This would be the location for authoritative doc on this feature.
        Hide
        stack added a comment -

        And v3 is the default now, in trunk at least.

        Show
        stack added a comment - And v3 is the default now, in trunk at least.
        Hide
        Andrew Purtell added a comment -

        It so happens I did a rebase last week. I also moved it into HFile V2 as there's nothing V3 specific about it. Let me check javadoc and improve the HCD docs and put up a new patch shortly.

        Show
        Andrew Purtell added a comment - It so happens I did a rebase last week. I also moved it into HFile V2 as there's nothing V3 specific about it. Let me check javadoc and improve the HCD docs and put up a new patch shortly.
        Hide
        Andrew Purtell added a comment -

        Attached updated patches.

        Show
        Andrew Purtell added a comment - Attached updated patches.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12645671/HBASE-9857-trunk.patch
        against trunk revision .
        ATTACHMENT ID: 12645671

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 26 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        -1 lineLengths. The patch introduces the following lines longer than 100:
        + public HFileReaderV3(final Path path, FixedFileTrailer trailer, final FSDataInputStreamWrapper fsdis,
        + family.setPrefetchBlocksOnOpen(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN)

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestHCM

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645671/HBASE-9857-trunk.patch against trunk revision . ATTACHMENT ID: 12645671 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 26 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 lineLengths . The patch introduces the following lines longer than 100: + public HFileReaderV3(final Path path, FixedFileTrailer trailer, final FSDataInputStreamWrapper fsdis, + family.setPrefetchBlocksOnOpen(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN) +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9547//console This message is automatically generated.
        Hide
        Andrew Purtell added a comment -

        TestHCM passes locally looped 10 times.

        Updated the trunk patch with one more Javadoc fix, I used the 0.98 tree to fix issues and trunk had an additional nit, a misplaced close tag for a @link.

        Show
        Andrew Purtell added a comment - TestHCM passes locally looped 10 times. Updated the trunk patch with one more Javadoc fix, I used the 0.98 tree to fix issues and trunk had an additional nit, a misplaced close tag for a @link.
        Hide
        Andrew Purtell added a comment -

        Looks like the new findbugs warning is from the TTL pretty printing patch: Result of integer multiplication cast to long in org.apache.hadoop.hbase.util.PrettyPrinter.humanReadableTTL(long)

        Show
        Andrew Purtell added a comment - Looks like the new findbugs warning is from the TTL pretty printing patch: Result of integer multiplication cast to long in org.apache.hadoop.hbase.util.PrettyPrinter.humanReadableTTL(long)
        Hide
        Andrew Purtell added a comment -

        Going to commit this to trunk and 0.98 this evening unless objection.

        Show
        Andrew Purtell added a comment - Going to commit this to trunk and 0.98 this evening unless objection.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12645814/HBASE-9857-trunk.patch
        against trunk revision .
        ATTACHMENT ID: 12645814

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 27 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        -1 lineLengths. The patch introduces the following lines longer than 100:
        + public HFileReaderV3(final Path path, FixedFileTrailer trailer, final FSDataInputStreamWrapper fsdis,
        + family.setPrefetchBlocksOnOpen(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN)

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestHCM

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645814/HBASE-9857-trunk.patch against trunk revision . ATTACHMENT ID: 12645814 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 27 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 lineLengths . The patch introduces the following lines longer than 100: + public HFileReaderV3(final Path path, FixedFileTrailer trailer, final FSDataInputStreamWrapper fsdis, + family.setPrefetchBlocksOnOpen(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN) +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9556//console This message is automatically generated.
        Hide
        ramkrishna.s.vasudevan added a comment -

        PrefetchExecutor misses the License header. May be rename it as HfileBlockPrefetchExecutor? (just a nit).
        On Exception during prefetch should cancel() be called on the catch block?

        Show
        ramkrishna.s.vasudevan added a comment - PrefetchExecutor misses the License header. May be rename it as HfileBlockPrefetchExecutor? (just a nit). On Exception during prefetch should cancel() be called on the catch block?
        Hide
        Andrew Purtell added a comment -

        Thanks for noticing the missing header. Will add it.

        I'm on the phone so don't have the patch in front of me. I think cancel() is not needed because we remove the entry for the prefetch upon exception but will double check tomorrow.

        Show
        Andrew Purtell added a comment - Thanks for noticing the missing header. Will add it. I'm on the phone so don't have the patch in front of me. I think cancel() is not needed because we remove the entry for the prefetch upon exception but will double check tomorrow.
        Hide
        Nick Dimiduk added a comment -

        This is looking very nice. I think we'll want to add a similar prefetch feature for index and bloom blocks, but that can come in a later revision.

        +1

        Show
        Nick Dimiduk added a comment - This is looking very nice. I think we'll want to add a similar prefetch feature for index and bloom blocks, but that can come in a later revision. +1
        Hide
        Andrew Purtell added a comment -

        The prefetch loads all blocks except meta blocks, so that includes indexes and blooms.

        Show
        Andrew Purtell added a comment - The prefetch loads all blocks except meta blocks, so that includes indexes and blooms.
        Hide
        Nick Dimiduk added a comment -

        Yes of course; I wasn't clear in my comment. I'm suggesting future work would enable only prefetching non-data blocks. Thus guaranteeing indices are always warm. One would want this when you know data size >> cache size.

        Show
        Nick Dimiduk added a comment - Yes of course; I wasn't clear in my comment. I'm suggesting future work would enable only prefetching non-data blocks. Thus guaranteeing indices are always warm. One would want this when you know data size >> cache size.
        Hide
        Andrew Purtell added a comment -

        Commit on hold until the SVN->GIT migration is finished.

        Show
        Andrew Purtell added a comment - Commit on hold until the SVN->GIT migration is finished.
        Hide
        Andrew Purtell added a comment -

        Ah right, sounds good Nick.

        Show
        Andrew Purtell added a comment - Ah right, sounds good Nick.
        Hide
        Andrew Purtell added a comment -

        Committed to trunk and 0.98. Thanks for the reviews!

        Show
        Andrew Purtell added a comment - Committed to trunk and 0.98. Thanks for the reviews!
        Hide
        Vladimir Rodionov added a comment -

        Andrew Purtell, do you take into account that all new blocks are cached in young gen space, which is 25% of overall cache? If you do not read block immediately after write (into cache) it will never get promoted into multi-bucket (50% of a cache) and you will be trashing bottom 25% of a block cache?

        Show
        Vladimir Rodionov added a comment - Andrew Purtell , do you take into account that all new blocks are cached in young gen space, which is 25% of overall cache? If you do not read block immediately after write (into cache) it will never get promoted into multi-bucket (50% of a cache) and you will be trashing bottom 25% of a block cache?
        Hide
        Andrew Purtell added a comment -

        If you do not read block immediately after write (into cache) it will never get promoted into multi-bucket (50% of a cache) and you will be trashing bottom 25% of a block cache

        We have a separate already existing schema setting for cache-on-write.

        Otherwise, sure, there's no magic here. It's a tuning option.

        Show
        Andrew Purtell added a comment - If you do not read block immediately after write (into cache) it will never get promoted into multi-bucket (50% of a cache) and you will be trashing bottom 25% of a block cache We have a separate already existing schema setting for cache-on-write. Otherwise, sure, there's no magic here. It's a tuning option.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #5137 (See https://builds.apache.org/job/HBase-TRUNK/5137/)
        HBASE-9857 Blockcache prefetch option (apurtell: rev 58818496daad0572843eacbeabfb95bc6af816ee)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
        • hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
        • hbase-shell/src/main/ruby/hbase/admin.rb
        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java
          Amend HBASE-9857 Blockcache prefetch option; add missing license header (apurtell: rev 264725d59274374d7b9c8ee2b47a86713ab1a6b8)
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5137 (See https://builds.apache.org/job/HBase-TRUNK/5137/ ) HBASE-9857 Blockcache prefetch option (apurtell: rev 58818496daad0572843eacbeabfb95bc6af816ee) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java hbase-shell/src/main/ruby/hbase/admin.rb hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java Amend HBASE-9857 Blockcache prefetch option; add missing license header (apurtell: rev 264725d59274374d7b9c8ee2b47a86713ab1a6b8) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98 #311 (See https://builds.apache.org/job/HBase-0.98/311/)
        HBASE-9857 Blockcache prefetch option (apurtell: rev 9b9f4df87a432778cc6f16becfeeea72e507e526)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java
        • hbase-shell/src/main/ruby/hbase/admin.rb
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          Amend HBASE-9857 Blockcache prefetch option; add missing license header (apurtell: rev f87fc3cea43698e9f417b12aabb7bd4f7932155a)
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98 #311 (See https://builds.apache.org/job/HBase-0.98/311/ ) HBASE-9857 Blockcache prefetch option (apurtell: rev 9b9f4df87a432778cc6f16becfeeea72e507e526) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java hbase-shell/src/main/ruby/hbase/admin.rb hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java Amend HBASE-9857 Blockcache prefetch option; add missing license header (apurtell: rev f87fc3cea43698e9f417b12aabb7bd4f7932155a) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #314 (See https://builds.apache.org/job/HBase-0.98/314/)
        Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev a260c862a71c6433213e6619c4de004250468c83)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #314 (See https://builds.apache.org/job/HBase-0.98/314/ ) Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev a260c862a71c6433213e6619c4de004250468c83) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #5140 (See https://builds.apache.org/job/HBase-TRUNK/5140/)
        Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev be85f89cd4508de5337820a2694d5262c6d69092)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5140 (See https://builds.apache.org/job/HBase-TRUNK/5140/ ) Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev be85f89cd4508de5337820a2694d5262c6d69092) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/)
        Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev a260c862a71c6433213e6619c4de004250468c83)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/ ) Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev a260c862a71c6433213e6619c4de004250468c83) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java
        Hide
        Enis Soztutar added a comment -

        Closing this issue after 0.99.0 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

          People

          • Assignee:
            Andrew Purtell
            Reporter:
            Andrew Purtell
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development