Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-5535

Replay delete markers during server side global index rebuild

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.0.0, 4.14.3
    • Fix Version/s: 4.15.0, 5.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently index rebuilds for global index tables are done on the server side. Phoenix client generates an aggregate plan using ServerBuildIndexCompiler to scan every data table row on the server side . This complier sets the scan attributes so that the row mutations that are scanned by UngroupedRegionObserver are then replayed on the data table so that index table rows are rebuilt. During this replay, data table row updates are skipped and only index table row are updated.

      Phoenix allows column entries to have null values. Null values are represented by HBase column delete marker. This means that index rebuild must replay these delete markers along with put mutations. In order to do that ServerBuildIndexCompiler should use raw scans but currently it does use regular scans. This leads incorrect index rebuilds when null values are used.

      A simple test where a data table with one global index with a covered column that can take null value is sufficient to reproduce this problem.

      1. Create a data table with columns  a,  b, and c where a is the primary key and c can have null value
      2. Write one row with not null values
      3. Overwrite the covered column with null (i.e., set it to null) 
      4. Create an index on the table where b is the secondary key and c is covered column
      5. Rebuild the index
      6. Dump the index table

      The index table row should have the null value for the covered column. However, it has the not null value written at step 2.  

        Attachments

        1. PHOENIX-5535.4.x-HBase-1.5.001.patch
          13 kB
          Kadir OZDEMIR
        2. PHOENIX-5535.master.001.patch
          12 kB
          Kadir OZDEMIR
        3. PHOENIX-5535.master.002.patch
          13 kB
          Kadir OZDEMIR
        4. PHOENIX-5535.master.003.patch
          40 kB
          Kadir OZDEMIR
        5. PHOENIX-5535.master.004.patch
          12 kB
          Kadir OZDEMIR

          Activity

            People

            • Assignee:
              kozdemir Kadir OZDEMIR
              Reporter:
              kozdemir Kadir OZDEMIR
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4.5h
                4.5h