Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18410 FilterList Improvement.
  3. HBASE-18879

HBase FilterList cause KeyOnlyFilter not work

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.4
    • Fix Version/s: None
    • Component/s: Filters
    • Labels:
      None
    • Environment:

      OS: Red Hat 4.4.7-11
      Hadoop: 2.6.4
      Hbase: 1.2.4

    • Hadoop Flags:
      Reviewed

      Description

      when use FilterList and KeyOnlyFilter together, if we put KeyOnlyFilter before FilterList, the KeyOnlyFilter may not work, means it will also grab the cell values:

      List<Filter> filters = new ArrayList<Filter>();
              Filter filter1 = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("column1"),
                      CompareOp.EQUAL, Bytes.toBytes("value1"));
              Filter filter2 = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("column1"),
                      CompareOp.EQUAL, Bytes.toBytes("value2"));
              filters.add(filter1);
              filters.add(filter2);
      
              FilterList filterListAll = new FilterList(Operator.MUST_PASS_ALL, 
                      new KeyOnlyFilter(),
                      new FilterList(Operator.MUST_PASS_ONE, filters));
      

      use the above code as filter to scan a table, it will return the cells with value instead of only return the key, if we put KeyOnlyFilter after FilterList as following, it works well.

      FilterList filterListAll = new FilterList(Operator.MUST_PASS_ALL,
                      new FilterList(Operator.MUST_PASS_ONE, filters),
                      new KeyOnlyFilter());
      

      the cause should due to the following code at hbase-client FilterList.java

      @Override
        @edu.umd.cs.findbugs.annotations.SuppressWarnings(value="SF_SWITCH_FALLTHROUGH",
          justification="Intentional")
        public ReturnCode filterKeyValue(Cell v) throws IOException {
          this.referenceKV = v;
      
          // Accumulates successive transformation of every filter that includes the Cell:
          Cell transformed = v;
      
          ReturnCode rc = operator == Operator.MUST_PASS_ONE?
              ReturnCode.SKIP: ReturnCode.INCLUDE;
          int listize = filters.size();
          for (int i = 0; i < listize; i++) {
            Filter filter = filters.get(i);
            if (operator == Operator.MUST_PASS_ALL) {
              if (filter.filterAllRemaining()) {
                return ReturnCode.NEXT_ROW;
              }
      LINE1      ReturnCode code = filter.filterKeyValue(v);{color}        
      switch (code) {
              // Override INCLUDE and continue to evaluate.
              case INCLUDE_AND_NEXT_COL:
                rc = ReturnCode.INCLUDE_AND_NEXT_COL; // FindBugs SF_SWITCH_FALLTHROUGH
              case INCLUDE:
      LINE2          transformed = filter.transformCell(transformed);{color}          
      continue;
              case SEEK_NEXT_USING_HINT:
                seekHintFilter = filter;
                return code;
              default:
                return code;
              }
            }
      

      notice the “LINE1”,"LINE2" , first line is a recursive invocation, it will assign a Cell results to the FilterList.transformedKV(we call it A), the results is from the FilterList with 2 SingleColumnValueFilter, so A with contains the cell value, while the second line with return A to the var transformed.
      back to the following loop, we can see the FilterList return results is var "transformed " which will override in each loop, so the value is determined by the last filter, so the order of KeyOnlyFilter will impact the results.

       Cell transformed = v;
      
          ReturnCode rc = operator == Operator.MUST_PASS_ONE?
              ReturnCode.SKIP: ReturnCode.INCLUDE;
          int listize = filters.size();
          for (int i = 0; i < listize; i++) {
            Filter filter = filters.get(i);
            if (operator == Operator.MUST_PASS_ALL) {
              if (filter.filterAllRemaining()) {
                return ReturnCode.NEXT_ROW;
              }
              ReturnCode code = filter.filterKeyValue(v);
              switch (code) {
              // Override INCLUDE and continue to evaluate.
              case INCLUDE_AND_NEXT_COL:
                rc = ReturnCode.INCLUDE_AND_NEXT_COL; // FindBugs SF_SWITCH_FALLTHROUGH
              case INCLUDE:
                transformed = filter.transformCell(transformed);
                continue;
              case SEEK_NEXT_USING_HINT:
                seekHintFilter = filter;
                return code;
              default:
                return code;
              }
            
      

        Attachments

        1. HBASE-18879-HBASE-18410.v2.patch
          9 kB
          Zheng Hu
        2. HBASE-18879-HBASE-18410.v1.patch
          8 kB
          Zheng Hu

          Issue Links

            Activity

              People

              • Assignee:
                openinx Zheng Hu
                Reporter:
                pengxu ZHA_Moonlight
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: