[HCATALOG-631] HBase e2e tests on single nodes on Hadoop 2.0.3 with "dfs.client.read.shortcircuit" turning on for HBase

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6
• Component/s: hbase
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 13/Mar/13 22:10
• Updated: 29/Aug/13 21:36
• Resolved: 28/Aug/13 21:54

Description

All HBase e2e fail: Pig_HBase_1, Pig_HBase_2.

Here is diagnosis:

• "dfs.client.read.shortcircuit" in hbase-site.xml is set to true, and "dfs.block.local-path-access.user" in hdfs-site.xml is set to "hbase"
• HCat is running as "hcat"
• HCat reads the hbase config and try to access hdfs block using local path
• "hcat" is not in "dfs.block.local-path-access.user", so local access fail

HDFS-4595 fix the issue in hdfs side. When local access fail, fall back to conventional read. However, performance wise, it is not desired because every time hdfs will try and fail while we know this will fail.

Another possible fix is on hbase side, which separate client/server config, since "dfs.client.read.shortcircuit" is only needed on server side.

Before hbase do anything, I provide a walk around to force "dfs.client.read.shortcircuit" to false on hcat side.