HBase
  1. HBase
  2. HBASE-7813

Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 0.94.6, 0.95.0
    • Fix Version/s: None
    • Component/s: Coprocessors
    • Labels:
      None

      Description

      Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and give it a set of Columns which are not in the row.

      A Delete is constructed using just the row alone. No family/column/version is added and it is applied, thus deleting the entire row.

      1. 7813-0.94.txt
        4 kB
        Varun Sharma

        Activity

        Hide
        Anoop Sam John added a comment -

        How is your Scan object looks like? You mean the Scan object contains some columns which are not at all present in row. Still the region scan returns some KVs for that row?

        Show
        Anoop Sam John added a comment - How is your Scan object looks like? You mean the Scan object contains some columns which are not at all present in row. Still the region scan returns some KVs for that row?
        Hide
        Varun Sharma added a comment -

        Code follow for BulkDeleteEndpoint:

        1) Delete delete = new Delete(row);
        2) Find stuff which is present and then do deleteColumn and deleteColumns and deleteFamily accordingly
        3) Apply delete

        If only 1) and 3) execute, entire row is deleted. If this makes sense, I have a patch which should resolve this, I am seeing this in my setup.

        Thanks !

        Show
        Varun Sharma added a comment - Code follow for BulkDeleteEndpoint: 1) Delete delete = new Delete(row); 2) Find stuff which is present and then do deleteColumn and deleteColumns and deleteFamily accordingly 3) Apply delete If only 1) and 3) execute, entire row is deleted. If this makes sense, I have a patch which should resolve this, I am seeing this in my setup. Thanks !
        Hide
        Ted Yu added a comment -

        I think before executing the deletion in step 3, we should check for row deletion.

        @Varun:
        Can you upload your patch ?

        Show
        Ted Yu added a comment - I think before executing the deletion in step 3, we should check for row deletion. @Varun: Can you upload your patch ?
        Hide
        Varun Sharma added a comment -

        Attaching for 0.94, can do trunk if this looks okay.

        Show
        Varun Sharma added a comment - Attaching for 0.94, can do trunk if this looks okay.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12568849/7813-0.94.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4412//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12568849/7813-0.94.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4412//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        When I ran unit test, I got:

        testBulkDeleteEndpoint(org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol)  Time elapsed: 40.614 sec  <<< ERROR!
        java.lang.reflect.UndeclaredThrowableException
          at $Proxy22.delete(Unknown Source)
          at org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol$1.call(TestBulkDeleteProtocol.java:124)
          at org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol$1.call(TestBulkDeleteProtocol.java:122)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1463)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:680)
        Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
        Mon Feb 11 09:33:35 PST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@40f186b8, java.io.IOException: java.io.IOException: java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Lorg.apache.hadoop.hbase.util.Pair;
        ...
          at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
          at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79)
          ... 9 more
        

        Do you see similar failure ?

        Show
        Ted Yu added a comment - When I ran unit test, I got: testBulkDeleteEndpoint(org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol) Time elapsed: 40.614 sec <<< ERROR! java.lang.reflect.UndeclaredThrowableException at $Proxy22.delete(Unknown Source) at org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol$1.call(TestBulkDeleteProtocol.java:124) at org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol$1.call(TestBulkDeleteProtocol.java:122) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1463) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:680) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions: Mon Feb 11 09:33:35 PST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@40f186b8, java.io.IOException: java.io.IOException: java.lang.ClassCastException: [Ljava.lang. Object ; cannot be cast to [Lorg.apache.hadoop.hbase.util.Pair; ... at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183) at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79) ... 9 more Do you see similar failure ?
        Hide
        Lars Hofhansl added a comment -

        Have you actually seen this fail? In createDeleteMutation I see this:

        case DeleteType.VERSION:
        ...
                  for (KeyValue kv : deleteRow) {
                    delete.deleteColumn(kv.getFamily(), kv.getQualifier(), kv.getTimestamp());
                    noOfVersionsToDelete++;
                  }
        ...
        

        So that should be doing the right thing.

        Show
        Lars Hofhansl added a comment - Have you actually seen this fail? In createDeleteMutation I see this: case DeleteType.VERSION: ... for (KeyValue kv : deleteRow) { delete.deleteColumn(kv.getFamily(), kv.getQualifier(), kv.getTimestamp()); noOfVersionsToDelete++; } ... So that should be doing the right thing.
        Hide
        Varun Sharma added a comment -

        Yes, but I am using DeleteType.COLUMN since I want all versions of a column to be deleted. In this particular case, I didn't say where we check that noOfVersions > 0 before applying the delete. Applying a delete without deleteColumn or deleteColumns deletes the entire row.

        Show
        Varun Sharma added a comment - Yes, but I am using DeleteType.COLUMN since I want all versions of a column to be deleted. In this particular case, I didn't say where we check that noOfVersions > 0 before applying the delete. Applying a delete without deleteColumn or deleteColumns deletes the entire row.
        Hide
        Varun Sharma added a comment -

        Hey Ted,

        Is this the correct way to the run test, its running zero tests for me...

        mvn test -Dtest=org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol

        Show
        Varun Sharma added a comment - Hey Ted, Is this the correct way to the run test, its running zero tests for me... mvn test -Dtest=org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol
        Hide
        Ted Yu added a comment -

        You need to specify '-PlocalTests ' on the command line.

        Show
        Ted Yu added a comment - You need to specify '-PlocalTests ' on the command line.
        Hide
        Varun Sharma added a comment -

        This maybe a red herring - lemme cross check this - we started losing rows after the BulkDeleteEndpoint but I now see the if (deleteRows.size() > 0) check, so I am still looking at why we are losing rows.

        Show
        Varun Sharma added a comment - This maybe a red herring - lemme cross check this - we started losing rows after the BulkDeleteEndpoint but I now see the if (deleteRows.size() > 0) check, so I am still looking at why we are losing rows.
        Hide
        Anoop Sam John added a comment -

        Varun Sharma Can you add your scenario into unit test and attach here?

        Show
        Anoop Sam John added a comment - Varun Sharma Can you add your scenario into unit test and attach here?
        Hide
        Anoop Sam John added a comment -

        Varun SharmaAny update on this? You found the actual issue?

        Show
        Anoop Sam John added a comment - Varun Sharma Any update on this? You found the actual issue?
        Hide
        Varun Sharma added a comment -

        Yeah, I don't think there was an issue here - we found a bug in the app logic instead, closing for now - may reopen if i see seomthing suspicious again

        Show
        Varun Sharma added a comment - Yeah, I don't think there was an issue here - we found a bug in the app logic instead, closing for now - may reopen if i see seomthing suspicious again

          People

          • Assignee:
            Varun Sharma
            Reporter:
            Varun Sharma
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development