HBase
  1. HBase
  2. HBASE-5837

hbase shell deleteall to .META. allows insertion of malformed rowkey

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.6, 0.95.2
    • Fix Version/s: 0.95.0
    • Component/s: master, shell
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When using the hbase shell to manipulate meta entries, one is allowed to 'delete' malformed rows (entries with less than 2 ascii 44 ',' chars). When this happens HBase servers may go down and the cluster will not be restartable without manual intervention.

      The delete results in a durable malformed rowkey in .META.'s memstore, .META.'s HLog, and eventually .META.'s HFiles. Subsequent scans to meta (such as when a HMaster starts) fail in the scanner because the comparator fails. In the case of an HMaster startup, it causes an abort that kills the HMaster process.

      12/04/18 22:07:34 FATAL master.HMaster: Unhandled exception. Starting shutdown.
      org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.IllegalArgumentException: No 44 in <blah,1334744821162.81f2df35c332dd2d3bb966fb5b419568.>, length=47, offset=54
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:990)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:979)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1894)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1834)
      at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
      Caused by: java.lang.IllegalArgumentException: No 44 in <blah,1334744821162.81f2df35c332dd2d3bb966fb5b419568.>, length=47, offset=54
      at org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.java:1300)
      at org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.java:1846)
      at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:130)
      at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:257)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2435)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2391)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2408)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1870)
      ... 6 more
      
      at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
      at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
      at $Proxy9.next(Unknown Source)
      at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:264)
      at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:237)
      at org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfResults(MetaReader.java:220)
      at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:1580)
      at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:221)
      at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:422)
      at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:295)
      12/04/18 22:07:34 INFO master.HMaster: Aborting 
      
      1. HBASE-5837.patch
        0.7 kB
        Ricky Saltzer

        Issue Links

          Activity

          Jonathan Hsieh created issue -
          Hide
          Jonathan Hsieh added a comment -

          Here's an example shell command that will leave you with a borked hbase:

          deleteall '.META.', 'table,<not a comma>'
          
          Show
          Jonathan Hsieh added a comment - Here's an example shell command that will leave you with a borked hbase: deleteall '.META.', 'table,<not a comma>'
          Hide
          Kevin Odell added a comment -

          I am working on a patch for this.

          Show
          Kevin Odell added a comment - I am working on a patch for this.
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Assignee Kevin Odell [ kevin.odell ]
          Kevin Odell made changes -
          Assignee Kevin Odell [ kevin.odell ]
          Ricky Saltzer made changes -
          Assignee Ricky Saltzer [ rickysaltzer ]
          Hide
          Ricky Saltzer added a comment -

          I've created a patch which adds checks to make sure a row exists before performing a delete on it.

          Show
          Ricky Saltzer added a comment - I've created a patch which adds checks to make sure a row exists before performing a delete on it.
          Ricky Saltzer made changes -
          Attachment HBASE-5837.patch [ 12548636 ]
          Hide
          Ricky Saltzer added a comment -

          I made a change in hbase-server/src/main/ruby/hbase/table.rb which checks to make sure a row exists before proceeding with deleting it. I've confirmed it works in standalone mode.

          Show
          Ricky Saltzer added a comment - I made a change in hbase-server/src/main/ruby/hbase/table.rb which checks to make sure a row exists before proceeding with deleting it. I've confirmed it works in standalone mode.
          Ricky Saltzer made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Hide
          Jimmy Xiang added a comment -

          +1 looks good to me.

          Show
          Jimmy Xiang added a comment - +1 looks good to me.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12548636/HBASE-5837.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 81 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548636/HBASE-5837.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 81 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3029//console This message is automatically generated.
          Ted Yu made changes -
          Summary hbase shell deleteall to .META. allows insertion of malformed rowkey. hbase shell deleteall to .META. allows insertion of malformed rowkey
          Hide
          Ted Yu added a comment -

          Integrated to trunk.

          Thanks for the patch, Ricky.

          Thanks for the review, Jimmy.

          Show
          Ted Yu added a comment - Integrated to trunk. Thanks for the patch, Ricky. Thanks for the review, Jimmy.
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #217 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/217/)
          HBASE-5837 hbase shell deleteall to .META. allows insertion of malformed rowkey (Ricky Saltzer) (Revision 1396857)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #217 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/217/ ) HBASE-5837 hbase shell deleteall to .META. allows insertion of malformed rowkey (Ricky Saltzer) (Revision 1396857) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3442 (See https://builds.apache.org/job/HBase-TRUNK/3442/)
          HBASE-5837 hbase shell deleteall to .META. allows insertion of malformed rowkey (Ricky Saltzer) (Revision 1396857)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3442 (See https://builds.apache.org/job/HBase-TRUNK/3442/ ) HBASE-5837 hbase shell deleteall to .META. allows insertion of malformed rowkey (Ricky Saltzer) (Revision 1396857) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Lars Hofhansl made changes -
          Link This issue relates to HBASE-7990 [ HBASE-7990 ]
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #879 (See https://builds.apache.org/job/HBase-0.94/879/)
          HBASE-7990 Backport HBASE-5837 'hbase shell deleteall to .META. allows insertion of malformed rowkey' to 0.94 (Revision 1452443)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/ruby/hbase/table.rb
          Show
          Hudson added a comment - Integrated in HBase-0.94 #879 (See https://builds.apache.org/job/HBase-0.94/879/ ) HBASE-7990 Backport HBASE-5837 'hbase shell deleteall to .META. allows insertion of malformed rowkey' to 0.94 (Revision 1452443) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/ruby/hbase/table.rb
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #12 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/12/)
          HBASE-7990 Backport HBASE-5837 'hbase shell deleteall to .META. allows insertion of malformed rowkey' to 0.94 (Revision 1452443)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/ruby/hbase/table.rb
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #12 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/12/ ) HBASE-7990 Backport HBASE-5837 'hbase shell deleteall to .META. allows insertion of malformed rowkey' to 0.94 (Revision 1452443) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/ruby/hbase/table.rb
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #116 (See https://builds.apache.org/job/HBase-0.94-security/116/)
          HBASE-7990 Backport HBASE-5837 'hbase shell deleteall to .META. allows insertion of malformed rowkey' to 0.94 (Revision 1452443)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/ruby/hbase/table.rb
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #116 (See https://builds.apache.org/job/HBase-0.94-security/116/ ) HBASE-7990 Backport HBASE-5837 'hbase shell deleteall to .META. allows insertion of malformed rowkey' to 0.94 (Revision 1452443) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/ruby/hbase/table.rb
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          Hide
          Huned Lokhandwala added a comment -

          Hi,

          On HBase 0.95 build, trying to add a delete command with a future timestamp for a row fails with errors as below. But on HBase 0.94 this works with no problems. This seems like a loss of backward compatibility with HBase 0.94.

          Commands below:

          []$ hbase version
          2013-08-12 20:26:33,468 INFO main util.VersionInfo: HBase 0.95.2.2.0.5.0-27

          []$ hbase shell
          2013-08-12 20:26:41,054 WARN main conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
          HBase Shell; enter 'help<RETURN>' for list of supported commands.
          Type "exit<RETURN>" to leave the HBase Shell
          Version 0.95.2.2.0.5.0-27, rUnknown, Sun Aug 11 07:31:44 PDT 2013

          hbase(main):003:0* create 'testtable', 'f1'
          0 row(s) in 1.8670 seconds
          => Hbase::Table - testtable

          hbase(main):004:0> exists 'testtable'
          Table testtable does exist
          0 row(s) in 0.0580 seconds

          hbase(main):006:0* put 'testtable', '1', 'f1:q1', 'value-1', 20
          0 row(s) in 0.0660 seconds

          hbase(main):007:0> delete 'testtable', '2', 'f1:q1', 9223372036854775806
          ERROR: Row Not Found
          Here is some help for this command:
          Put a delete cell value at specified table/row/column and optionally
          timestamp coordinates. Deletes must match the deleted cell's
          coordinates exactly. When scanning, a delete cell suppresses older
          versions. To delete a cell from 't1' at row 'r1' under column 'c1'
          marked with the time 'ts1', do:
          hbase> delete 't1', 'r1', 'c1', ts1
          The same command can also be run on a table reference. Suppose you had a reference
          t to table 't1', the corresponding command would be:
          hbase> t.delete 'r1', 'c1', ts1

          Show
          Huned Lokhandwala added a comment - Hi, On HBase 0.95 build, trying to add a delete command with a future timestamp for a row fails with errors as below. But on HBase 0.94 this works with no problems. This seems like a loss of backward compatibility with HBase 0.94. Commands below: []$ hbase version 2013-08-12 20:26:33,468 INFO main util.VersionInfo: HBase 0.95.2.2.0.5.0-27 []$ hbase shell 2013-08-12 20:26:41,054 WARN main conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 0.95.2.2.0.5.0-27, rUnknown, Sun Aug 11 07:31:44 PDT 2013 hbase(main):003:0* create 'testtable', 'f1' 0 row(s) in 1.8670 seconds => Hbase::Table - testtable hbase(main):004:0> exists 'testtable' Table testtable does exist 0 row(s) in 0.0580 seconds hbase(main):006:0* put 'testtable', '1', 'f1:q1', 'value-1', 20 0 row(s) in 0.0660 seconds hbase(main):007:0> delete 'testtable', '2', 'f1:q1', 9223372036854775806 ERROR: Row Not Found Here is some help for this command: Put a delete cell value at specified table/row/column and optionally timestamp coordinates. Deletes must match the deleted cell's coordinates exactly. When scanning, a delete cell suppresses older versions. To delete a cell from 't1' at row 'r1' under column 'c1' marked with the time 'ts1', do: hbase> delete 't1', 'r1', 'c1', ts1 The same command can also be run on a table reference. Suppose you had a reference t to table 't1', the corresponding command would be: hbase> t.delete 'r1', 'c1', ts1
          Hide
          Huned Lokhandwala added a comment -

          The use case for the above is that, suppose I want to place a future delete marker to prevent a put command in the future, such that:

          hbase(main):007:0> delete 'testtable', '2', 'f1:q1', 9223372036854775806

          hbase(main):006:0* put 'testtable', '2', 'f1:q1', 'value-2', 30

          would prevent the row 2 from being added to the 'testtable'. this functionality was supported in hbase 0.94 but is now lost in hbase 0.95.

          Show
          Huned Lokhandwala added a comment - The use case for the above is that, suppose I want to place a future delete marker to prevent a put command in the future, such that: hbase(main):007:0> delete 'testtable', '2', 'f1:q1', 9223372036854775806 hbase(main):006:0* put 'testtable', '2', 'f1:q1', 'value-2', 30 would prevent the row 2 from being added to the 'testtable'. this functionality was supported in hbase 0.94 but is now lost in hbase 0.95.
          Hide
          Jonathan Hsieh added a comment -

          I personally don't think deleting something that doesn't exist in the future makes sense.

          How do you detect that someone did this? It would seem like data loss or a denial of service. If I write in the present and there is a delete in the future, my impression is that hbase must be losing data!

          That said - can you provide evidence that someone actually depends on this behavior in a real world situation?

          Show
          Jonathan Hsieh added a comment - I personally don't think deleting something that doesn't exist in the future makes sense. How do you detect that someone did this? It would seem like data loss or a denial of service. If I write in the present and there is a delete in the future, my impression is that hbase must be losing data! That said - can you provide evidence that someone actually depends on this behavior in a real world situation?
          Hide
          Ted Yu added a comment -

          The patch covered ruby script. What about Java client ?

          Show
          Ted Yu added a comment - The patch covered ruby script. What about Java client ?
          Hide
          Enis Soztutar added a comment -

          This patch causes divergence between Java client and shell. Deleting a non-existing row IS an HBase feature, it should not be disabled from shell. Agreed that this might not be used much (and increasingly confusing), I think we should continue supporting it.

          Having said that, we can address the root cause of this issue in a different way. I think we should make these kind of sanity checks in the server side (maybe a permanent META coprocessor?)

          Show
          Enis Soztutar added a comment - This patch causes divergence between Java client and shell. Deleting a non-existing row IS an HBase feature, it should not be disabled from shell. Agreed that this might not be used much (and increasingly confusing), I think we should continue supporting it. Having said that, we can address the root cause of this issue in a different way. I think we should make these kind of sanity checks in the server side (maybe a permanent META coprocessor?)
          Hide
          Enis Soztutar added a comment -

          Should we revert this one, and open another issue?

          Show
          Enis Soztutar added a comment - Should we revert this one, and open another issue?
          Hide
          Jonathan Hsieh added a comment -

          I detailed the original problem – which was only concerned about future deletes in META and feel that this should be illegal for META.

          Instead of reverting and then opening a new issue and then a fix eventually happening, can we file a new issue, and revert when the server side fix for this goes in?

          Show
          Jonathan Hsieh added a comment - I detailed the original problem – which was only concerned about future deletes in META and feel that this should be illegal for META. Instead of reverting and then opening a new issue and then a fix eventually happening, can we file a new issue, and revert when the server side fix for this goes in?
          Hide
          Jonathan Hsieh added a comment -

          Or at the least limit the current patch to attempts at changes applied to META.

          Show
          Jonathan Hsieh added a comment - Or at the least limit the current patch to attempts at changes applied to META.
          Hide
          stack added a comment -

          Marking closed.

          Show
          stack added a comment - Marking closed.
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Ricky Saltzer
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development