Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3139

Failure in map-reduce job due to undefined hdp.version variable when using HDP stack and remote HBase cluster

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: v2.2.0
    • Fix Version/s: v2.3.0
    • Component/s: Others
    • Labels:
    • Environment:
      HDP 2.5.6, two cluster setup, Kylin 2.2.0 in a cluster with Hive only, remote HBase cluster for data storage

      Description

      When running on top of HDP stack and using a setup where Hive and HBase run in different clusters cube build/refresh fails on the step "Extract Fact Table Distinct Columns" with the error

      java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path

      Based on existing JIRA discussions in Ambari project, it's responsibility of a service to set hdp.version Java property. When HBase is not installed as a service in a cluster where Kylin server is running, hbase launcher (invoked by kylin.sh) does not set this property (presumably because HBase in that case is just a client and not a service).
      The only suitable workaround found so far is to set property as part of the conf/setenv.sh script.

      In order to avoid hard coding of the HDP version info, suggested change to setenv.sh will attempt to detect HDP version at run-time. It should work for all released HDP version from 2.2.x to 2.6.x
      In addition to that, it will also try to locate and set Java native library path, when running on top of HDP.

        Attachments

        1. KYLIN-3139.master.001.patch
          2 kB
          Vsevolod Ostapenko

          Activity

            People

            • Assignee:
              seva_ostapenko Vsevolod Ostapenko
              Reporter:
              seva_ostapenko Vsevolod Ostapenko
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified