Uploaded image for project: 'HCatalog'
  1. HCatalog
  2. HCATALOG-631

HBase e2e tests on single nodes on Hadoop 2.0.3 with "dfs.client.read.shortcircuit" turning on for HBase

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6
    • Component/s: hbase
    • Labels:
      None

      Description

      All HBase e2e fail: Pig_HBase_1, Pig_HBase_2.

      Here is diagnosis:

      • "dfs.client.read.shortcircuit" in hbase-site.xml is set to true, and "dfs.block.local-path-access.user" in hdfs-site.xml is set to "hbase"
      • HCat is running as "hcat"
      • HCat reads the hbase config and try to access hdfs block using local path
      • "hcat" is not in "dfs.block.local-path-access.user", so local access fail

      HDFS-4595 fix the issue in hdfs side. When local access fail, fall back to conventional read. However, performance wise, it is not desired because every time hdfs will try and fail while we know this will fail.

      Another possible fix is on hbase side, which separate client/server config, since "dfs.client.read.shortcircuit" is only needed on server side.

      Before hbase do anything, I provide a walk around to force "dfs.client.read.shortcircuit" to false on hcat side.

        Attachments

        1. HCATALOG-631-1.patch
          0.8 kB
          Jianyong Dai

          Activity

            People

            • Assignee:
              daijy Jianyong Dai
              Reporter:
              daijy Jianyong Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: