[KYLIN-3139] Failure in map-reduce job due to undefined hdp.version variable when using HDP stack and remote HBase cluster

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: v2.2.0
• Fix Version/s: v2.3.0
• Component/s: Others
• Labels: hdp
• Environment: HDP 2.5.6, two cluster setup, Kylin 2.2.0 in a cluster with Hive only, remote HBase cluster for data storage
• Assignee: Vsevolod Ostapenko
• Reporter: Vsevolod Ostapenko
• Created: 28/Dec/17 21:17
• Updated: 02/Apr/18 13:42
• Resolved: 09/Feb/18 01:14

Description

When running on top of HDP stack and using a setup where Hive and HBase run in different clusters cube build/refresh fails on the step "Extract Fact Table Distinct Columns" with the error

java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path

Based on existing JIRA discussions in Ambari project, it's responsibility of a service to set hdp.version Java property. When HBase is not installed as a service in a cluster where Kylin server is running, hbase launcher (invoked by kylin.sh) does not set this property (presumably because HBase in that case is just a client and not a service).
The only suitable workaround found so far is to set property as part of the conf/setenv.sh script.

In order to avoid hard coding of the HDP version info, suggested change to setenv.sh will attempt to detect HDP version at run-time. It should work for all released HDP version from 2.2.x to 2.6.x
In addition to that, it will also try to locate and set Java native library path, when running on top of HDP.