HBase
  1. HBase
  2. HBASE-5257

Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.3, 0.95.0
    • Component/s: Filters, Performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are various usecases and filter types where evaluating the filter before version are handled either do not make sense, or make filter handling more complicated.

      Also see this comment in ScanQueryMatcher:

          /**
           * Filters should be checked before checking column trackers. If we do
           * otherwise, as was previously being done, ColumnTracker may increment its
           * counter for even that KV which may be discarded later on by Filter. This
           * would lead to incorrect results in certain cases.
           */
      

      So we had Filters after the column trackers (which do the version checking), and then moved it.
      Should be at the discretion of the Filter.
      Could either add a new method to FilterBase (maybe excludeVersions() or something). Or have a new Filter wrapper (like WhileMatchFilter), that should only be used as outmost filter and indicates the same (maybe ExcludeVersionsFilter).

      See latest comments on HBASE-5229 for motivation.

      1. 5257-0.92.addendum
        0.6 kB
        Ted Yu
      2. HBASE-5257-0.94.txt
        10 kB
        Varun Sharma
      3. HBASE-5257-0.92.txt
        10 kB
        Varun Sharma
      4. 5257-trunk-v2.txt
        9 kB
        Ted Yu
      5. 5257-trunk.txt
        8 kB
        Ted Yu

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          @Lars:
          The latest comments (or a summary) from HBASE-5229 should be copied in this JIRA.

          Show
          Ted Yu added a comment - @Lars: The latest comments (or a summary) from HBASE-5229 should be copied in this JIRA.
          Hide
          Lars Hofhansl added a comment -

          @Ted: Linked the issues instead.

          As for this issue... For maximum flexibility and to avoid introducing wire incompatibility I propose a small code change in ScanQueryMatcher and a new VersionFilterWrapper that takes two Filters (both of course can the FilterLists), the first is evaluated pre column tracker, the 2nd is run post column tracker.

          Show
          Lars Hofhansl added a comment - @Ted: Linked the issues instead. As for this issue... For maximum flexibility and to avoid introducing wire incompatibility I propose a small code change in ScanQueryMatcher and a new VersionFilterWrapper that takes two Filters (both of course can the FilterLists), the first is evaluated pre column tracker, the 2nd is run post column tracker.
          Hide
          Lars Hofhansl added a comment -

          Running filter post column trackers does only work for Filters that do nothing in filterRowKey and filterRow.

          Show
          Lars Hofhansl added a comment - Running filter post column trackers does only work for Filters that do nothing in filterRowKey and filterRow.
          Hide
          Lars Hofhansl added a comment -

          As this comes up in mailing list frequently, let's revive this.

          Show
          Lars Hofhansl added a comment - As this comes up in mailing list frequently, let's revive this.
          Hide
          ramkrishna.s.vasudevan added a comment -

          +1. I think some new JIRA also was raised for this.
          I think filters like ColumnCountGetFilter or ColumnPagniation filter can be grouped to new filter type and may be if the filterKeyValue of such filters return INCLUDE then the ColumnTrackers should be considering this and then decide on filtering?

          Show
          ramkrishna.s.vasudevan added a comment - +1. I think some new JIRA also was raised for this. I think filters like ColumnCountGetFilter or ColumnPagniation filter can be grouped to new filter type and may be if the filterKeyValue of such filters return INCLUDE then the ColumnTrackers should be considering this and then decide on filtering?
          Hide
          Lars Hofhansl added a comment -

          Would ColumnCountGetFilter or ColumnPagniationFilter never be evaluated before the version tracking? What if they are included in a FilterList together with other filters? I think that would quickly become inscrutable.

          We could:

          1. Say that these filters can never be included in a FilterList (and maybe that is best approach as it makes little sense anyway)
          2. Do What I suggest above. Have a special filter wrapper that takes exactly two filters (which can be FilterLists and each of which can be null) and evaluate one before the version counting and one after. This wrapper must always be at the top level of a filter, it cannot be included in a FilterList

          When I attempted a patch on this a while back, the details quickly became non-trivial.

          Thoughts?

          Show
          Lars Hofhansl added a comment - Would ColumnCountGetFilter or ColumnPagniationFilter never be evaluated before the version tracking? What if they are included in a FilterList together with other filters? I think that would quickly become inscrutable. We could: Say that these filters can never be included in a FilterList (and maybe that is best approach as it makes little sense anyway) Do What I suggest above. Have a special filter wrapper that takes exactly two filters (which can be FilterLists and each of which can be null) and evaluate one before the version counting and one after. This wrapper must always be at the top level of a filter, it cannot be included in a FilterList When I attempted a patch on this a while back, the details quickly became non-trivial. Thoughts?
          Hide
          Ted Yu added a comment -

          Through either annotation or introduction of special interfaces, we can distinguish the two types of filters (w.r.t. timing of version tracking).
          Different types of filters should not coexist in the same FilterList.

          My two cents.

          Show
          Ted Yu added a comment - Through either annotation or introduction of special interfaces, we can distinguish the two types of filters (w.r.t. timing of version tracking). Different types of filters should not coexist in the same FilterList. My two cents.
          Hide
          Lars Hofhansl added a comment -

          Yep. We still need to support a single scan to specify before and after filter, though. Otherwise there is no way to combine these in a scan.

          Show
          Lars Hofhansl added a comment - Yep. We still need to support a single scan to specify before and after filter, though. Otherwise there is no way to combine these in a scan.
          Hide
          ramkrishna.s.vasudevan added a comment -

          2nd approach seems to be fine with me Lars.

          Show
          ramkrishna.s.vasudevan added a comment - 2nd approach seems to be fine with me Lars.
          Hide
          Lars Hofhansl added a comment - - edited

          Ted's approach might be easier to grasp, though.

          I'm curious, do we think there are filters for which it never makes sense to evaluate them before the version tracking - and vice versa?

          Show
          Lars Hofhansl added a comment - - edited Ted's approach might be easier to grasp, though. I'm curious, do we think there are filters for which it never makes sense to evaluate them before the version tracking - and vice versa?
          Hide
          Varun Sharma added a comment -

          Currently, ColumnCountGetFilter and ColumnPaginationFilter suffer from this issue - they always undercount when there are multiple versions of a cell (even when max versions of a column family is set to 1 - I think this is because the versions exist until compaction happens). I looked at the ScanQueryMatcher/StoreScanner/ColumnTracker code and it seems that there is one other plausible approach towards resolving this. Currently, if a filter wants to skip over a KeyValue pair, it has 2 options - skip to next key value pair which could be the same column (SKIP) or skip to next column (SEEK_NEXT_COL). Though we are providing the filters a mechanism to really skip in these two ways when they exclude the value, we don't do that when they "include" the value. The INCLUDE always causes a seek to the next key value pair. I think that probably makes sense for the ColumnTracker since for column tracking we never want to seek across columns after doing an INCLUDE but for filters we probably want symmetry when trying to INCLUDE/EXCLUDE key value pairs. So, I was proposing something like:

          1) Introduce INCLUDE_AND_SEEK_NEXT_COL to Filter.ReturnCode
          2) Introduce INCLUDE_AND_SEEK_NEXT_COL to ScanQueryMatcher.MatchCode
          3) Modify StoreScanner accordingly to seek to next column after the include and also link the above two types in the match() function
          4) Finally modify ColumnPaginationFilter to return SEEK_NEXT_COL,INCLUDE_AND_SEEK_NEXT_COL instead of SKIP,INCLUDE_AND_SEEK_NEXT_COL respectively. Similarly for ColumnCountGetFilter

          This might be a more direct way of resolving this issue and would avoid the column tracker sandwich between two layers of filters. What do you think, lars ?

          Varun

          Show
          Varun Sharma added a comment - Currently, ColumnCountGetFilter and ColumnPaginationFilter suffer from this issue - they always undercount when there are multiple versions of a cell (even when max versions of a column family is set to 1 - I think this is because the versions exist until compaction happens). I looked at the ScanQueryMatcher/StoreScanner/ColumnTracker code and it seems that there is one other plausible approach towards resolving this. Currently, if a filter wants to skip over a KeyValue pair, it has 2 options - skip to next key value pair which could be the same column (SKIP) or skip to next column (SEEK_NEXT_COL). Though we are providing the filters a mechanism to really skip in these two ways when they exclude the value, we don't do that when they "include" the value. The INCLUDE always causes a seek to the next key value pair. I think that probably makes sense for the ColumnTracker since for column tracking we never want to seek across columns after doing an INCLUDE but for filters we probably want symmetry when trying to INCLUDE/EXCLUDE key value pairs. So, I was proposing something like: 1) Introduce INCLUDE_AND_SEEK_NEXT_COL to Filter.ReturnCode 2) Introduce INCLUDE_AND_SEEK_NEXT_COL to ScanQueryMatcher.MatchCode 3) Modify StoreScanner accordingly to seek to next column after the include and also link the above two types in the match() function 4) Finally modify ColumnPaginationFilter to return SEEK_NEXT_COL,INCLUDE_AND_SEEK_NEXT_COL instead of SKIP,INCLUDE_AND_SEEK_NEXT_COL respectively. Similarly for ColumnCountGetFilter This might be a more direct way of resolving this issue and would avoid the column tracker sandwich between two layers of filters. What do you think, lars ? Varun
          Hide
          Varun Sharma added a comment -

          Hey guys,

          Sorry but I did not see the work log and the earlier proposal(s) - I am new to JIRA but regarding my above thought - that is more close to the "excludeVersions()" option mentioned in the issue - since versions are discarded in this approach. Btw, we would also need one change to FilterList.filterKeyValue() where INCLUDE_AND_SEEK_NEXT_COL would override INCLUDE so if one filter returned INCLUDE and another INCLUDE_AND_SEEK_NEXT_COL, the result is INCLUDE_AND_SEEK_NEXT_COL - this would mean that we can mix ColumnPaginationFilter/ColumnCountGetFilter with other filters but we would only get back the latest column versions. I doubt if there is a compelling use case for counting/pagination of versions... my 2 cents.

          Thanks !
          Varun

          Show
          Varun Sharma added a comment - Hey guys, Sorry but I did not see the work log and the earlier proposal(s) - I am new to JIRA but regarding my above thought - that is more close to the "excludeVersions()" option mentioned in the issue - since versions are discarded in this approach. Btw, we would also need one change to FilterList.filterKeyValue() where INCLUDE_AND_SEEK_NEXT_COL would override INCLUDE so if one filter returned INCLUDE and another INCLUDE_AND_SEEK_NEXT_COL, the result is INCLUDE_AND_SEEK_NEXT_COL - this would mean that we can mix ColumnPaginationFilter/ColumnCountGetFilter with other filters but we would only get back the latest column versions. I doubt if there is a compelling use case for counting/pagination of versions... my 2 cents. Thanks ! Varun
          Hide
          Ted Yu added a comment -

          4) Finally modify ColumnPaginationFilter to return SEEK_NEXT_COL,INCLUDE_AND_SEEK_NEXT_COL instead of SKIP,INCLUDE_AND_SEEK_NEXT_COL respectively. Similarly for ColumnCountGetFilter

          I guess you meant this:
          4) Finally modify ColumnPaginationFilter to return SEEK_NEXT_COL,INCLUDE_AND_SEEK_NEXT_COL instead of SKIP,SEEK_NEXT_COL respectively. Similarly for ColumnCountGetFilter

          @Varun:
          If you can provide a patch as you outlined above, that would be nice.

          Show
          Ted Yu added a comment - 4) Finally modify ColumnPaginationFilter to return SEEK_NEXT_COL,INCLUDE_AND_SEEK_NEXT_COL instead of SKIP,INCLUDE_AND_SEEK_NEXT_COL respectively. Similarly for ColumnCountGetFilter I guess you meant this: 4) Finally modify ColumnPaginationFilter to return SEEK_NEXT_COL,INCLUDE_AND_SEEK_NEXT_COL instead of SKIP,SEEK_NEXT_COL respectively. Similarly for ColumnCountGetFilter @Varun: If you can provide a patch as you outlined above, that would be nice.
          Hide
          Varun Sharma added a comment -

          Sure, I will put together a patch and submit as soon as its ready. Thanks !

          Show
          Varun Sharma added a comment - Sure, I will put together a patch and submit as soon as its ready. Thanks !
          Hide
          Varun Sharma added a comment -

          Attached patch passed unittests for Hbase 0.92 - hbase 0.95-snapshot will need a different patch.

          Show
          Varun Sharma added a comment - Attached patch passed unittests for Hbase 0.92 - hbase 0.95-snapshot will need a different patch.
          Hide
          Olson,Andrew added a comment -

          I will be out of the office with limited access to email until Monday, 10/29/2012. For urgent issues please contact Greg Whitsitt.

          Andrew Olson | Sr. Software Architect | Cerner Corporation | 816.201.3825 | aolson1@cerner.com | www.cerner.com

          Show
          Olson,Andrew added a comment - I will be out of the office with limited access to email until Monday, 10/29/2012. For urgent issues please contact Greg Whitsitt. Andrew Olson | Sr. Software Architect | Cerner Corporation | 816.201.3825 | aolson1@cerner.com | www.cerner.com
          Hide
          Ted Yu added a comment -

          Patch for trunk.

          TestColumnPaginationFilter, TestFilter and TestFilterList passed.

          Show
          Ted Yu added a comment - Patch for trunk. TestColumnPaginationFilter, TestFilter and TestFilterList passed.
          Hide
          Varun Sharma added a comment -

          Ted Yu

          Thanks for patching this against 0.96 - should we also be submitting into 0.92/0.94 ? We are using the 0.92 version of hbase ?

          Show
          Varun Sharma added a comment - Ted Yu Thanks for patching this against 0.96 - should we also be submitting into 0.92/0.94 ? We are using the 0.92 version of hbase ?
          Hide
          Varun Sharma added a comment -

          Patch for 0.94

          Show
          Varun Sharma added a comment - Patch for 0.94
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550721/HBASE-5257-0.94.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3141//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550721/HBASE-5257-0.94.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3141//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550711/5257-trunk.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 82 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550711/5257-trunk.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 82 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3140//console This message is automatically generated.
          Hide
          Varun Sharma added a comment -

          Corrected patch file in right format for 0.92

          Show
          Varun Sharma added a comment - Corrected patch file in right format for 0.92
          Hide
          Varun Sharma added a comment -

          Correctly formatted patch file...

          Show
          Varun Sharma added a comment - Correctly formatted patch file...
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550728/HBASE-5257-0.94.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3142//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550728/HBASE-5257-0.94.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3142//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          I like the patch. It is a simple yet effective solution for this, which can also be used by other filters in the future.

          Show
          Lars Hofhansl added a comment - I like the patch. It is a simple yet effective solution for this, which can also be used by other filters in the future.
          Hide
          Lars Hofhansl added a comment -

          We should also handle ColumnCountGetFilter.

          Show
          Lars Hofhansl added a comment - We should also handle ColumnCountGetFilter.
          Hide
          Lars Hofhansl added a comment -

          I'm good on 0.94.
          @Ted: Your call for 0.92

          Show
          Lars Hofhansl added a comment - I'm good on 0.94. @Ted: Your call for 0.92
          Hide
          Lars Hofhansl added a comment -

          I added you as contributor, Varun, and assigned this issue to you.

          Show
          Lars Hofhansl added a comment - I added you as contributor, Varun, and assigned this issue to you.
          Hide
          Lars Hofhansl added a comment -

          The changes to ScanQueryMatcher abd FilterList look good.

          +1 on patch.

          Show
          Lars Hofhansl added a comment - The changes to ScanQueryMatcher abd FilterList look good. +1 on patch.
          Hide
          Lars Hofhansl added a comment -

          (after we also tackle ColumnCountGetFilter that is)

          Show
          Lars Hofhansl added a comment - (after we also tackle ColumnCountGetFilter that is)
          Hide
          Ted Yu added a comment -

          Patch v2 for trunk adds ColumnCountGetFilter, as Lars suggested.

          Varun:
          Please press 'Cancel Patch' before attaching patches for 0.94 / 0.92 so that Hadoop QA doesn't test them.

          Show
          Ted Yu added a comment - Patch v2 for trunk adds ColumnCountGetFilter, as Lars suggested. Varun: Please press 'Cancel Patch' before attaching patches for 0.94 / 0.92 so that Hadoop QA doesn't test them.
          Hide
          Lars Hofhansl added a comment -

          +1 on v2.
          Thanks Ted.

          Show
          Lars Hofhansl added a comment - +1 on v2. Thanks Ted.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550741/5257-trunk-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 82 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550741/5257-trunk-v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 82 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3144//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          You don't want to fix this for 0.94 Arun? (or did you remove the 0.94.3 by accident?)

          Show
          Lars Hofhansl added a comment - You don't want to fix this for 0.94 Arun? (or did you remove the 0.94.3 by accident?)
          Hide
          Varun Sharma added a comment -

          Actually, I did not have the ColumnCountGetFilter changes in the previous patches - so I removed the earlier patches and made the changes and now added back with the new changes...

          Show
          Varun Sharma added a comment - Actually, I did not have the ColumnCountGetFilter changes in the previous patches - so I removed the earlier patches and made the changes and now added back with the new changes...
          Hide
          Lars Hofhansl added a comment -

          Cool. It's typically better to just attach new patches and name them xyx_v2, etc. So that comments referring to earlier patches still make sense.

          Show
          Lars Hofhansl added a comment - Cool. It's typically better to just attach new patches and name them xyx_v2, etc. So that comments referring to earlier patches still make sense.
          Hide
          ramkrishna.s.vasudevan added a comment - - edited

          @Lars/@Ted/varun
          First of all thanks for the patch.
          I have some doubts here
          Versions are cell based right?
          Now with this change even if the maxVersions is set as 2, after the first cell is included we jump to the second col itself.
          So if col qualifier 'q1' has two versions still we move on to colqualifier 'q2'? Am i missing something ?

          {Edit}

          : removed 'family' word.

          Show
          ramkrishna.s.vasudevan added a comment - - edited @Lars/@Ted/ varun First of all thanks for the patch. I have some doubts here Versions are cell based right? Now with this change even if the maxVersions is set as 2, after the first cell is included we jump to the second col itself. So if col qualifier 'q1' has two versions still we move on to colqualifier 'q2'? Am i missing something ? {Edit} : removed 'family' word.
          Hide
          Lars Hofhansl added a comment -

          @Ram: That is the idea ColumnPaginationFilter and ColumnCountGetFilter are interested in columns (not versions).
          So moving on to q2 after we see the first version of q1 is exactly what we want... Maybe I am missing something...?

          Show
          Lars Hofhansl added a comment - @Ram: That is the idea ColumnPaginationFilter and ColumnCountGetFilter are interested in columns (not versions). So moving on to q2 after we see the first version of q1 is exactly what we want... Maybe I am missing something...?
          Hide
          ramkrishna.s.vasudevan added a comment -

          Okie...i think i was wrong ...

          Show
          ramkrishna.s.vasudevan added a comment - Okie...i think i was wrong ...
          Hide
          Ted Yu added a comment -

          Integrated to 0.92, 0.94 and trunk.

          Thanks for the patch, Varun.

          Thanks for the review, Lars and Ram.

          Show
          Ted Yu added a comment - Integrated to 0.92, 0.94 and trunk. Thanks for the patch, Varun. Thanks for the review, Lars and Ram.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #599 (See https://builds.apache.org/job/HBase-0.92/599/)
          HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402211)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #599 (See https://builds.apache.org/job/HBase-0.92/599/ ) HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402211) Result = FAILURE tedyu : Files : /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/Filter.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Hide
          Ted Yu added a comment -

          Addendum for 0.92, fixing a typo.

          Show
          Ted Yu added a comment - Addendum for 0.92, fixing a typo.
          Hide
          Ted Yu added a comment -

          Addendum integrated to 0.92

          Show
          Ted Yu added a comment - Addendum integrated to 0.92
          Hide
          Varun Sharma added a comment -

          Thanks Lars, Ram and Ted and sorry for the few hiccups while submitting patches...

          Should I resolve this now ?

          Show
          Varun Sharma added a comment - Thanks Lars, Ram and Ted and sorry for the few hiccups while submitting patches... Should I resolve this now ?
          Hide
          Lars Hofhansl added a comment -

          Committed to 0.92, 0.94, and 0.96... Resolving.

          Thanks for the patch Varun!

          Show
          Lars Hofhansl added a comment - Committed to 0.92, 0.94, and 0.96... Resolving. Thanks for the patch Varun!
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #554 (See https://builds.apache.org/job/HBase-0.94/554/)
          HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402210)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #554 (See https://builds.apache.org/job/HBase-0.94/554/ ) HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402210) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/Filter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3486 (See https://builds.apache.org/job/HBase-TRUNK/3486/)
          HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402209)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3486 (See https://builds.apache.org/job/HBase-TRUNK/3486/ ) HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402209) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/Filter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #236 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/)
          HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402209)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #236 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/ ) HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402209) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/Filter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #600 (See https://builds.apache.org/job/HBase-0.92/600/)
          HBASE-5257 Addendum fixes typo in ColumnCountGetFilter.java (Revision 1402225)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #600 (See https://builds.apache.org/job/HBase-0.92/600/ ) HBASE-5257 Addendum fixes typo in ColumnCountGetFilter.java (Revision 1402225) Result = FAILURE tedyu : Files : /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/)
          HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402210)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/ ) HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402210) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/Filter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92-security #146 (See https://builds.apache.org/job/HBase-0.92-security/146/)
          HBASE-5257 Addendum fixes typo in ColumnCountGetFilter.java (Revision 1402225)
          HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402211)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java

          tedyu :
          Files :

          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
          Show
          Hudson added a comment - Integrated in HBase-0.92-security #146 (See https://builds.apache.org/job/HBase-0.92-security/146/ ) HBASE-5257 Addendum fixes typo in ColumnCountGetFilter.java (Revision 1402225) HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter (Varun) (Revision 1402211) Result = FAILURE tedyu : Files : /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java tedyu : Files : /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/Filter.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java

            People

            • Assignee:
              Varun Sharma
              Reporter:
              Lars Hofhansl
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development