Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18620

Secure bulkload job fails when HDFS umask has limited scope

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: security
    • Labels:
      None

      Description

      By default "hbase.fs.tmp.dir" parameter value is /user/${user.name}/hbase-staging.

      RegionServer creates the staging directory (hbase.bulkload.staging.dir, default value is hbase.fs.tmp.dir) during opening a region as below when SecureBulkLoadEndpoint configured in hbase.coprocessor.region.classes,

      drwx------ - hbase hadoop 0 2017-08-12 13:55 /user/xyz
      drwx--x--x - hbase hadoop 0 2017-08-12 13:55 /user/xyz/hbase-staging
      drwx--x--x - hbase hadoop 0 2017-08-12 13:55 /user/xyz/hbase-staging/DONOTERASE
      

      Here,
      1. RegionServer is started using "xyz" linux user.
      2. HDFS umask (fs.permissions.umask-mode) has been set as 077, so file/dir permission will not be wider than 700. "/user/xyz" directory (doesn't exist earlier) permission will be 700 and "/user/xyz/hbase-staging" will be 711 as we are just setting permission of staging directory not the parent directories which are created (fs.mkdirs()) by RegionServer.

      Secure bulkload will fail as other user doesn't have EXECUTE permission on "/user/xyz" directory.

      Steps to reproduce:
      ==================
      1. Configure org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint in "hbase.coprocessor.region.classes" at client side.
      2. Login to machine as "root" linux user.
      3. kinit to any kerberos user except RegionServer kerberos user (say admin).
      4. ImportTSV will create the user temp directory (hbase.fs.tmp.dir) while writing partition file,

      drwxrwxrwx - admin hadoop 0 2017-08-12 14:52 /user/root
      drwxrwxrwx - admin hadoop 0 2017-08-12 14:52 /user/root/hbase-staging
      

      4. During LoadIncrementalHFiles job,

      • a. prepareBulkLoad() step - Random dir will be created by RegionServer credentials,
        drwxrwxrwx - hbase hadoop 0 2017-08-12 14:58 /user/xyz/hbase-staging/hbase__t1__e67b23m2ghe6fkn1bqrb95ak41ferj8957cdhsep4ebmpohm22nvi54vh8g3qh1
        
      • b. secureBulkLoadHFiles() step - Family dir existence check and creation is done by using client user credentials. Here client operation will fail as below,
        Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=admin, access=EXECUTE, inode="/user/xyz/hbase-staging/admin__t1__e1f3m4r2prud9117thg5pdg91lkg0le0fdvtbbpg03epqg0f14lv54j8sqd8s0n6/cf1":hbase:hadoop:drwx------
        	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:342)
        	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:279)
        	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:223)
        	at com.huawei.hadoop.adapter.hdfs.plugin.HWAccessControlEnforce.checkPermission(HWAccessControlEnforce.java:69)
        

      So the root cause is "admin" user doesn't have EXECUTE permission over "/user/xyz", because RegionServer has created this intermediate parent directory during opening (SecureBulkLoadEndpoint) a region where the default permission is set as 700 based on the hdfs UMASK 077.

      Solution:
      =========
      However it can be handled by the creating /user/xyz manually and setting sufficient permission explicitly. But we should handle this by setting sufficient permission to intermediate staging directories which is created by RegionServer.

        Attachments

        1. HBASE-18620-branch-1.patch
          3 kB
          Pankaj Kumar
        2. HBASE-18620-branch-1-v2.patch
          3 kB
          Pankaj Kumar
        3. HBASE-18620-branch-1-v3.patch
          3 kB
          Pankaj Kumar

          Activity

            People

            • Assignee:
              pankaj2461 Pankaj Kumar
              Reporter:
              pankaj2461 Pankaj Kumar
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: