HBase
  1. HBase
  2. HBASE-8626

RowMutations fail when Delete and Put on same columnFamily/column/row

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.94.7, 0.95.0
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None
    • Environment:

      Ubuntu 12.04, HBase 0.94.7

      Description

      When RowMutations have a Delete followed by Put to same column family or columns or rows, only the Delete is happening while the Put is ignored so atomicity of RowMutations is broken for such cases.

      Attached is a unit test where the following tests are failing:

      • testDeleteCFThenPutInSameCF: Delete a column family and then Put to same column family.
      • testDeleteColumnThenPutSameColumn: Delete a column and then Put to same column.
      • testDeleteRowThenPutSameRow: Delete a row and then Put to same row
      1. tests_for_row_mutations1.patch
        11 kB
        Vinod
      2. TestRowMutations.java
        10 kB
        Vinod
      3. 8626-v1.txt
        12 kB
        Ted Yu

        Activity

        Vinod created issue -
        Vinod made changes -
        Field Original Value New Value
        Attachment TestRowMutations.java [ 12584871 ]
        Vinod made changes -
        Description When RowMutations have a Delete followed by Put to same column family or columns or rows, only the Delete is happening while the Put is ignored so atomicity of RowMutations is broken for such cases.

        Attached is a unit test where the following tests are failing:
        - testDeleteCFThenPutInSameCF: Delete a column family and then Put to same column family.
        - testDeleteColumnThenPutSameColumn: Delete a column and then Put to same column.
        - testDeleteRowThenPutSameRow: Delete a row and then Put to same row


        ------------------
        package org.apache.hadoop.hbase.regionserver;

        import org.apache.hadoop.conf.Configuration;
        import org.apache.hadoop.fs.Path;
        import org.apache.hadoop.hbase.*;
        import org.apache.hadoop.hbase.client.*;
        import org.apache.hadoop.hbase.util.Bytes;
        import org.apache.hadoop.hbase.util.EnvironmentEdgeManagerTestHelper;
        import org.junit.experimental.categories.Category;

        import java.io.IOException;
        import java.util.NavigableMap;

        /**
         * Tests for {@link RowMutations} where we combine various scenarios of Delete followed by Put and vice versa.
         */
        @Category(MediumTests.class)
        public class TestRowMutations extends HBaseTestCase {

            HRegion region = null;

            private HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
            private final String DIR = TEST_UTIL.getDataTestDir("TestAtomicOperation").toString();

            static final byte[] tableName = Bytes.toBytes("TRM_testtable");
            static final byte [] row = Bytes.toBytes("rowA");
            static final byte[] cf1 = Bytes.toBytes("cf1");
            static final byte[] cf2 = Bytes.toBytes("cf2");
            static final byte[] qual1 = Bytes.toBytes("qual1");
            static final byte[] qual2 = Bytes.toBytes("qual2");
            static final byte[] value1 = Bytes.toBytes("value1");
            static final byte[] value2 = Bytes.toBytes("value2");


            /**
             * Test {@link RowMutations} where we Delete a column family and then Put to same column family. Fails currently.
             * @throws IOException
             */
            public void testDeleteCFThenPutInSameCF() throws IOException {
                initHRegion(tableName, getName(), HBaseConfiguration.create(), cf1);

                //delete entire column family
                Delete delete = new Delete(row);
                delete.deleteFamily(cf1);

                //Put to same column family
                Put put = new Put(row);
                put.add(cf1, qual1, value1);
                put.add(cf1, qual2, value2);

                RowMutations rm = new RowMutations(row);
                rm.add(delete);
                rm.add(put);
                region.mutateRow(rm);


                Result result = region.get(new Get(row));
                NavigableMap<byte[],byte[]> familyMap = result.getFamilyMap(cf1);
                assertNotNull(familyMap);
                assertEquals(2, familyMap.size());
                assertEquals(value1, familyMap.get(qual1));
                assertEquals(value2, familyMap.get(qual2));

            }

            /**
             * Test {@link RowMutations} where we Delete a column and then Put to same column. Fails currently
             * @throws IOException
             */
            public void testDeleteColumnThenPutSameColumn() throws IOException {
                initHRegion(tableName, getName(), HBaseConfiguration.create(), cf1);

                //delete columns
                Delete delete = new Delete(row);
                delete.deleteColumn(cf1, qual1);
                delete.deleteColumn(cf1, qual2);

                //Put to same columns
                Put put = new Put(row);
                put.add(cf1, qual1, value1);
                put.add(cf1, qual2, value2);

                RowMutations rm = new RowMutations(row);
                rm.add(delete);
                rm.add(put);
                region.mutateRow(rm);


                Result result = region.get(new Get(row));
                NavigableMap<byte[],byte[]> familyMap = result.getFamilyMap(cf1);
                assertNotNull(familyMap);
                assertEquals(2, familyMap.size());
                assertEquals(value1, familyMap.get(qual1));
                assertEquals(value2, familyMap.get(qual2));

            }

            /**
             * Test {@link RowMutations} where we Delete a row and then Put to same row. Fails currently.
             * @throws IOException
             */
            public void testDeleteRowThenPutSameRow() throws IOException {
                initHRegion(tableName, getName(), HBaseConfiguration.create(), cf1);

                //delete row
                Delete delete = new Delete(row);

                //Put to same row
                Put put = new Put(row);
                put.add(cf1, qual1, value1);
                put.add(cf1, qual2, value2);

                RowMutations rm = new RowMutations(row);
                rm.add(delete);
                rm.add(put);
                region.mutateRow(rm);


                Result result = region.get(new Get(row));
                NavigableMap<byte[],byte[]> familyMap = result.getFamilyMap(cf1);
                assertNotNull(familyMap);
                assertEquals(2, familyMap.size());
                assertEquals(value1, familyMap.get(qual1));
                assertEquals(value2, familyMap.get(qual2));

            }


            /**
             * Test {@link RowMutations} where we Put to a column family and then Delete same column family.
             * @throws IOException
             */
            public void testPutCFThenDeleteSameCF() throws IOException {
                initHRegion(tableName, getName(), HBaseConfiguration.create(), cf1);

                //Put to some column family
                Put put = new Put(row);
                put.add(cf1, qual1, value1);
                put.add(cf1, qual2, value2);

                //delete same column family
                Delete delete = new Delete(row);
                delete.deleteFamily(cf1);


                RowMutations rm = new RowMutations(row);
                rm.add(put);
                rm.add(delete);
                region.mutateRow(rm);


                Result result = region.get(new Get(row));
                NavigableMap<byte[],byte[]> familyMap = result.getFamilyMap(cf1);
                assertNull(familyMap);

            }

            /**
             * Test {@link RowMutations} where we Put to a column and then Delete same column.
             * @throws IOException
             */
            public void testPutToColumnThenDeleteSameColumn() throws IOException {
                initHRegion(tableName, getName(), HBaseConfiguration.create(), cf1);

                //Put to some columns
                Put put = new Put(row);
                put.add(cf1, qual1, value1);
                put.add(cf1, qual2, value2);

                //delete same columns
                Delete delete = new Delete(row);
                delete.deleteColumn(cf1, qual1);
                delete.deleteColumn(cf1, qual2);

                RowMutations rm = new RowMutations(row);
                rm.add(put);
                rm.add(delete);

                region.mutateRow(rm);


                Result result = region.get(new Get(row));
                NavigableMap<byte[],byte[]> familyMap = result.getFamilyMap(cf1);
                assertNull(familyMap);
            }

            /**
             * Test {@link RowMutations} where we Put to a row and then Delete same row.
             * @throws IOException
             */
            public void testPutRowThenDeleteSameRow() throws IOException {
                initHRegion(tableName, getName(), HBaseConfiguration.create(), cf1);

                //delete row
                Delete delete = new Delete(row);

                //Put to same row
                Put put = new Put(row);
                put.add(cf1, qual1, value1);
                put.add(cf1, qual2, value2);

                RowMutations rm = new RowMutations(row);
                rm.add(put);
                rm.add(delete);
                region.mutateRow(rm);


                Result result = region.get(new Get(row));
                NavigableMap<byte[],byte[]> familyMap = result.getFamilyMap(cf1);
                assertNull(familyMap);

            }

            /**
             * Test {@link RowMutations} where we Delete a column family and then Put to different column family. Passes currently.
             * @throws IOException
             */
            public void testDeleteCFThenPutInDiffCF() throws IOException {
                initHRegion(tableName, getName(), HBaseConfiguration.create(), cf1, cf2);

                //delete some column family
                Delete delete = new Delete(row);
                delete.deleteFamily(cf1);

                //Put to different column family
                Put put = new Put(row);
                put.add(cf2, qual1, value1);
                put.add(cf2, qual2, value2);

                RowMutations rm = new RowMutations(row);
                rm.add(delete);
                rm.add(put);
                region.mutateRow(rm);


                Result result = region.get(new Get(row));
                NavigableMap<byte[],byte[]> familyMap = result.getFamilyMap(cf2);
                assertNotNull(familyMap);
                assertEquals(2, familyMap.size());
                assertEquals(value1, familyMap.get(qual1));
                assertEquals(value2, familyMap.get(qual2));

                NavigableMap<byte[],byte[]> familyMap1 = result.getFamilyMap(cf1);
                assertEquals(0, familyMap1.size()); //TODO: why is familyMap1 empty instead of null?

            }

            /**
             * Test {@link RowMutations} where we Delete a column and then Put to different column. Passes currently.
             * @throws IOException
             */
            public void testDeleteColumnThenPutDiffColumn() throws IOException {
                initHRegion(tableName, getName(), HBaseConfiguration.create(), cf1, cf2);

                //delete columns
                Delete delete = new Delete(row);
                delete.deleteColumn(cf1, qual1);

                //Put to different column, one in same column family and one in another column family
                Put put = new Put(row);
                put.add(cf1, qual2, value2);
                put.add(cf2, qual2, value2);

                RowMutations rm = new RowMutations(row);
                rm.add(delete);
                rm.add(put);
                region.mutateRow(rm);


                Result result = region.get(new Get(row));
                NavigableMap<byte[],byte[]> familyMap = result.getFamilyMap(cf1);
                assertNotNull(familyMap);
                assertEquals(1, familyMap.size());
                assertEquals(value2, familyMap.get(qual2));

                NavigableMap<byte[],byte[]> familyMap2 = result.getFamilyMap(cf2);
                assertNotNull(familyMap2);
                assertEquals(1, familyMap2.size());
                assertEquals(value2, familyMap2.get(qual2));

            }



            private void initHRegion (byte [] tableName, String callingMethod,
                                      Configuration conf, byte [] ... families)
                    throws IOException{
                HTableDescriptor htd = new HTableDescriptor(tableName);
                for(byte [] family : families) {
                    htd.addFamily(new HColumnDescriptor(family));
                }
                HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false);
                Path path = new Path(DIR + callingMethod);
                if (fs.exists(path)) {
                    if (!fs.delete(path, true)) {
                        throw new IOException("Failed delete of " + path);
                    }
                }
                region = HRegion.createHRegion(info, path, conf, htd);
            }

            @Override
            protected void setUp() throws Exception {
                super.setUp();
            }

            @Override
            protected void tearDown() throws Exception {
                super.tearDown();
                EnvironmentEdgeManagerTestHelper.reset();
            }
        }

        ----------------



        When RowMutations have a Delete followed by Put to same column family or columns or rows, only the Delete is happening while the Put is ignored so atomicity of RowMutations is broken for such cases.

        Attached is a unit test where the following tests are failing:
        - testDeleteCFThenPutInSameCF: Delete a column family and then Put to same column family.
        - testDeleteColumnThenPutSameColumn: Delete a column and then Put to same column.
        - testDeleteRowThenPutSameRow: Delete a row and then Put to same row
        Vinod made changes -
        Affects Version/s 0.95.0 [ 12324094 ]
        Fix Version/s 0.95.1 [ 12324288 ]
        Vinod made changes -
        Attachment tests_for_row_mutations1.patch [ 12584875 ]
        Ted Yu made changes -
        Attachment 8626-v1.txt [ 12584879 ]
        Ted Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Ted Yu [ yuzhihong@gmail.com ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.7 [ 12324039 ]
        Ted Yu made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.95.1 [ 12324288 ]

          People

          • Assignee:
            Ted Yu
            Reporter:
            Vinod
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development