[HBASE-17861] Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0, 1.3.3, 1.2.7
• Labels: filesystem s3 wal
• Hadoop Flags: Reviewed
• Release Note: Some object store does not support unix style permission, this jira fix the permission check issue when specify staging dir in different file system. Currently it covers s3, wasb, swift.
• Assignee: Yi Liang
• Reporter: Yi Liang
• Created: 31/Mar/17 20:18
• Updated: 20/Aug/18 09:57
• Resolved: 04/Apr/17 17:40

Description

Found some issue, when set up HBASE-17437: Support specifying a WAL directory outside of the root directory.

The region server are showdown when I add following config into hbase-site.xml
hbase.rootdir = s3a://xx//xx
hbase.wal.dir = hdfs://xx/xx
hbase.coprocessor.region.classes = org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
Error is below

org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw java.lang.IllegalStateException: Directory already exists but permissions aren't set to '-rwx--x--x'

The reason is that, when hbase enable securebulkload, hbase will create a folder in s3, it can not set above permission, because in s3, all files are listed as having full read/write permissions and all directories appear to have full rwx permissions. See Object stores have differerent authorization models in https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html

Issue Links

• is related to: HBASE-17878 java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter when starting HBase with hbase.rootdir on S3
• relates to: HBASE-21060 fix dead store in SecureBulkLoadEndpoint