Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11591

Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.99.0, 2.0.0
    • Fix Version/s: 0.99.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      See discussion in HBASE-11339.
      When we have a case where there are same KVs in two files one produced by flush/compaction and the other thro the bulk load.
      Both the files have some same kvs which matches even in timestamp.
      Steps:
      Add some rows with a specific timestamp and flush the same.
      Bulk load a file with the same data.. Enusre that "assign seqnum" property is set.
      The bulk load should use HFileOutputFormat2 (or ensure that we write the bulk_time_output key).
      This would ensure that the bulk loaded file has the highest seq num.
      Assume the cell in the flushed/compacted store file is
      row1,cf,cq,ts1, value1 and the cell in the bulk loaded file is
      row1,cf,cq,ts1,value2
      (There are no parallel scans).
      Issue a scan on the table in 0.96. The retrieved value is row1,cf1,cq,ts1,value2
      But the same in 0.98 will retrieve row1,cf1,cq,ts2,value1.
      This is a behaviour change. This is because of this code

          public int compare(KeyValueScanner left, KeyValueScanner right) {
            int comparison = compare(left.peek(), right.peek());
            if (comparison != 0) {
              return comparison;
            } else {
              // Since both the keys are exactly the same, we break the tie in favor
              // of the key which came latest.
              long leftSequenceID = left.getSequenceID();
              long rightSequenceID = right.getSequenceID();
              if (leftSequenceID > rightSequenceID) {
                return -1;
              } else if (leftSequenceID < rightSequenceID) {
                return 1;
              } else {
                return 0;
              }
            }
          }
      

      Here in 0.96 case the mvcc of the cell in both the files will have 0 and so the comparison will happen from the else condition . Where the seq id of the bulk loaded file is greater and would sort out first ensuring that the scan happens from that bulk loaded file.
      In case of 0.98+ as we are retaining the mvcc+seqid we are not making the mvcc as 0 (remains a non zero positive value). Hence the compare() sorts out the cell in the flushed/compacted file. Which means though we know the lateset file is the bulk loaded file we don't scan the data.
      Seems to be a behaviour change. Will check on other corner cases also but we are trying to know the behaviour of bulk load because we are evaluating if it can be used for MOB design.

        Attachments

        1. TestBulkload.java
          6 kB
          ramkrishna.s.vasudevan
        2. HBASE-11591.patch
          15 kB
          ramkrishna.s.vasudevan
        3. HBASE-11591_1.patch
          16 kB
          ramkrishna.s.vasudevan
        4. HBASE-11591_2.patch
          16 kB
          ramkrishna.s.vasudevan
        5. HBASE-11591_3.patch
          17 kB
          ramkrishna.s.vasudevan
        6. hbase-11591-03-jeff.patch
          14 kB
          Jeffrey Zhong
        7. hbase-11591-03-02.patch
          15 kB
          Jeffrey Zhong
        8. HBASE-11591_5.patch
          22 kB
          ramkrishna.s.vasudevan
        9. HBASE-11591_6.patch
          22 kB
          ramkrishna.s.vasudevan
        10. HBASE-11591_6.patch
          22 kB
          ramkrishna.s.vasudevan
        11. HBASE-11591_branch-1-addendum.patch
          3 kB
          ramkrishna.s.vasudevan

          Issue Links

            Activity

              People

              • Assignee:
                ram_krish ramkrishna.s.vasudevan
                Reporter:
                ram_krish ramkrishna.s.vasudevan
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: