[HBASE-20898] Improve support for HDFS at-rest encryption

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Component/s: encryption

Environment

HBase 2 on Hadoop 2.6.0+ (HDFS at-rest encryption)

• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 17/Jul/18 00:38
• Updated: 25/Jul/18 16:38

Description

• Note * this has nothing to do with HBase's Transparent Encryption of Data At Rest.

HDFS's at rest encryption is "transparent" in that encrypt/decrypt itself doesn't require client side change. However, in practice, there re a few cases that need to be taken care of. For example, accessing KMS requires KMS delegation tokens. If HBase tools get only HDFS delegation tokens, it would fail to access files in HDFS encryption zone. Cases such as HBASE-20403 suggests in some cases HBase behaves differently in HDFS-encrypted cluster.

I propose an umbrella jira to revisit the HDFS at-rest encryption support in various HBase subcomponents and tools, add additional tests and enhance the tools as we visit them.

Issue Links

• relates to: HBASE-16178 HBase restore command fails on cluster with encrypted HDFS

Sub-Tasks

• Asyncfs should retry upon RetryStartFileException: Open
• Add Hadoop KMS dependency and basic HDFS at-rest encryption tests: Patch Available
• Improve FsDelegationToken to support KMS delegation tokens: Open
• Helper method to configure secure DFS cluster for tests: Resolved