HBase
  1. HBase
  2. HBASE-5714

Add write permissions check before any hbck run that modifies hdfs.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.6, 0.92.2, 0.94.0, 0.95.2
    • Fix Version/s: 0.94.2
    • Component/s: hbck
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We encoutered a situation where hbck was run by an under-privileged user that was unable to write/modify/merge regions due to hdfs perms. Unfortunately, this user was alerted of this after several minutes of read-only operations. hbck should fail early by having a write perm check and providing actionable advice to the hbase admin.

      Maybe something like: "Current user yy does not have write perms to <hbase home>. Please run hbck as hdfs user xxx"

      1. HBASE-5628.patch
        7 kB
        Liang Xie
      2. HBASE-5628.patch.v2
        7 kB
        Liang Xie
      3. hbase-5714-90.patch
        8 kB
        Jonathan Hsieh
      4. hbase-5714-92.patch
        7 kB
        Jonathan Hsieh
      5. hbase-5714-94.patch
        7 kB
        Jonathan Hsieh

        Issue Links

          Activity

          stack made changes -
          Fix Version/s 0.94.2 [ 12321884 ]
          Fix Version/s 0.95.0 [ 12324094 ]
          Hide
          stack added a comment -

          Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl)

          Show
          stack added a comment - Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl)
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          Fix Version/s 0.94.2 [ 12321884 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.2 [ 12321884 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.90.7 [ 12319481 ]
          Fix Version/s 0.92.2 [ 12319888 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.2 [ 12321884 ]
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/)
          HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375228)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/ ) HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375228) Result = FAILURE jmhsieh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92-security #116 (See https://builds.apache.org/job/HBase-0.92-security/116/)
          HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375229)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Show
          Hudson added a comment - Integrated in HBase-0.92-security #116 (See https://builds.apache.org/job/HBase-0.92-security/116/ ) HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375229) Result = FAILURE jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #48 (See https://builds.apache.org/job/HBase-0.94-security/48/)
          HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375228)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #48 (See https://builds.apache.org/job/HBase-0.94-security/48/ ) HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375228) Result = FAILURE jmhsieh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #509 (See https://builds.apache.org/job/HBase-0.92/509/)
          HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375229)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #509 (See https://builds.apache.org/job/HBase-0.92/509/ ) HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375229) Result = FAILURE jmhsieh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #139 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/139/)
          HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375227)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #139 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/139/ ) HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375227) Result = FAILURE jmhsieh : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #409 (See https://builds.apache.org/job/HBase-0.94/409/)
          HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375228)

          Result = SUCCESS
          jmhsieh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #409 (See https://builds.apache.org/job/HBase-0.94/409/ ) HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375228) Result = SUCCESS jmhsieh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3245 (See https://builds.apache.org/job/HBase-TRUNK/3245/)
          HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375227)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3245 (See https://builds.apache.org/job/HBase-TRUNK/3245/ ) HBASE-5714 Add write permissions check before any hbck run that modifies hdfs (Liang Xie) (Revision 1375227) Result = FAILURE jmhsieh : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          Jonathan Hsieh made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.90.7 [ 12319481 ]
          Fix Version/s 0.92.2 [ 12319888 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.2 [ 12321884 ]
          Resolution Fixed [ 1 ]
          Jonathan Hsieh made changes -
          Attachment hbase-5714-90.patch [ 12541656 ]
          Attachment hbase-5714-92.patch [ 12541657 ]
          Attachment hbase-5714-94.patch [ 12541658 ]
          Hide
          Jonathan Hsieh added a comment -

          Diffs of the backports.

          Show
          Jonathan Hsieh added a comment - Diffs of the backports.
          Jonathan Hsieh made changes -
          Assignee liang xie [ xieliang007 ]
          Jonathan Hsieh made changes -
          Parent HBASE-5628 [ 12547917 ]
          Issue Type Sub-task [ 7 ] Improvement [ 4 ]
          Hide
          Jonathan Hsieh added a comment -

          Thanks liang xie! Looks good to me. I'm committing (with minor tweaks to port to 0.94/0.92/0.90). Thanks for the review stack.

          Show
          Jonathan Hsieh added a comment - Thanks liang xie! Looks good to me. I'm committing (with minor tweaks to port to 0.94/0.92/0.90). Thanks for the review stack.
          Liang Xie made changes -
          Attachment HBASE-5628.patch.v2 [ 12541602 ]
          Hide
          Liang Xie added a comment -

          Modified per Jon's comments

          Show
          Liang Xie added a comment - Modified per Jon's comments
          Hide
          Jonathan Hsieh added a comment -

          Jie, looks great and thanks for the testing output. I have only a one style nit i'd like your thoughts on:

          Can you make the calls to Runtime.getRuntime().exit(xx) happen in main instead of the helper? If we were to write tests, the exit would make it difficult.

          Show
          Jonathan Hsieh added a comment - Jie, looks great and thanks for the testing output. I have only a one style nit i'd like your thoughts on: Can you make the calls to Runtime.getRuntime().exit(xx) happen in main instead of the helper? If we were to write tests, the exit would make it difficult.
          Hide
          stack added a comment -

          Patch looks good to me. Nice one Lliang. Jon or Jimmy want to take a look?

          Show
          stack added a comment - Patch looks good to me. Nice one Lliang. Jon or Jimmy want to take a look?
          Hide
          Liang Xie added a comment -

          Sorry for no new test-case attached,i'm still a very newbie for HBase project
          I tested as following steps:
          1) run as an under-privileged user on base code, issue this command : "hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair", it failed after several read-only operations

          2) apply this patch, rerun , got the fast-fail as expected :
          12/08/18 23:37:21 WARN util.HBaseFsck: Got AccessControlException when preCheckPermission
          org.apache.hadoop.security.AccessControlException: Permission denied: action=WRITE path=hdfs://10.235.2.121:9000/hbase/ROOT user=xiaomi
          at org.apache.hadoop.hbase.util.FSUtils.checkAccess(FSUtils.java:1064)
          at org.apache.hadoop.hbase.util.HBaseFsck.preCheckPermission(HBaseFsck.java:1184)
          at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:97)
          Current user xiaomi does not have write perms to hdfs://10.235.2.121:9000/hbase/ROOT. Please rerun hbck as hdfs user root

          Show
          Liang Xie added a comment - Sorry for no new test-case attached,i'm still a very newbie for HBase project I tested as following steps: 1) run as an under-privileged user on base code, issue this command : "hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair", it failed after several read-only operations 2) apply this patch, rerun , got the fast-fail as expected : 12/08/18 23:37:21 WARN util.HBaseFsck: Got AccessControlException when preCheckPermission org.apache.hadoop.security.AccessControlException: Permission denied: action=WRITE path=hdfs://10.235.2.121:9000/hbase/ ROOT user=xiaomi at org.apache.hadoop.hbase.util.FSUtils.checkAccess(FSUtils.java:1064) at org.apache.hadoop.hbase.util.HBaseFsck.preCheckPermission(HBaseFsck.java:1184) at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:97) Current user xiaomi does not have write perms to hdfs://10.235.2.121:9000/hbase/ ROOT . Please rerun hbck as hdfs user root
          Liang Xie made changes -
          Attachment HBASE-5628.patch [ 12541483 ]
          Hide
          Liang Xie added a comment -

          A patch for trunk version

          Show
          Liang Xie added a comment - A patch for trunk version
          Jonathan Hsieh made changes -
          Link This issue is duplicated by HBASE-5820 [ HBASE-5820 ]
          Jonathan Hsieh made changes -
          Link This issue is related to HBASE-5734 [ HBASE-5734 ]
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Description We encoutered a situation where hbase was run by a different user and was unable to write/modify/merge regions due to hdfs perms. Unfortunately, this happened after several minutes of read-only operations. hbck should fail early by having a write perm check and providing actionable advice to the hbase admin.

          Maybe something like: "Current user yy does not have write perms to <hbase home>. Please run hbck as hdfs user xxx"
          We encoutered a situation where hbck was run by an under-privileged user that was unable to write/modify/merge regions due to hdfs perms. Unfortunately, this user was alerted of this after several minutes of read-only operations. hbck should fail early by having a write perm check and providing actionable advice to the hbase admin.

          Maybe something like: "Current user yy does not have write perms to <hbase home>. Please run hbck as hdfs user xxx"
          Jonathan Hsieh created issue -

            People

            • Assignee:
              Liang Xie
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development