[HBASE-24111] Enable CompactionTool executions on non-HDFS filesystems

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.2.5
• Component/s: Compaction, mapreduce, tooling
• Assignee: Peter Somogyi
• Reporter: Peter Somogyi
• Created: 03/Apr/20 10:47
• Updated: 14/Jul/20 21:31
• Resolved: 06/Apr/20 10:31

Description

CompactionTool on non-HDFS filesystems does not work because the used filesystems are mixed up. To collect the StoreFiles the Filesystem.get(conf) method is used instead of relying on the root dir filesystem.

With YARN the delegation tokens are not handled correctly with a different filesystem and the mappers only get delegation token for the staging directory's filesystem.

Another issue is in MapReduce mode when YARN is used. In this case the TMP directory (hbase.tmp.dir) is created under yarn.nodemanager.local-dirs but this directory is created (even on HDFS a regular user can't create this directory) on the Storefile's filesystem where a local filesystem temp directory shouldn't be used.

In this ticket I plan to clean up the used filesystems, remove the custom temp directory (use the regions' .tmp directory) and collect delegation tokens for staging dir + store dir paths' FileSystems.

Issue Links

• links to: GitHub Pull Request #1427