[AMBARI-23286] check_fs_root() in hive_service.py may replace Hive HDFS locations with ":8020:8020"

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.1
• Fix Version/s: 2.6.1
• Component/s: ambari-server
• Assignee: Mahadev Konar
• Reporter: Jason Golieb
• Created: 19/Mar/18 17:28
• Updated: 19/Mar/18 17:28

Description

Modify check_fs_root() in hive_services.py to check if "fs_root" and "out" are not endswith('/'), append '/'

 

Background

When hive's metatstore database table 'DBS' contains a HDFS location which doesn't contain any port number, Ambari may incorrectly replace the HDFS locations.

For example, "hdfs://sandbox-hdp.hortonworks.com/apps/hive/warehouse/dummies.db" should be a valid location:

[hive@sandbox-hdp ~]$ hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -listFSRoot 2>/dev/null
Initializing HiveMetaTool..
Listing FS Roots..
hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db
hdfs://sandbox-hdp.hortonworks.com/apps/hive/warehouse/dummies.db
hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db
hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse

Ambari uses same command as below:

[hive@sandbox-hdp ~]$ hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -listFSRoot 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://sandbox-hdp.hortonworks.com:8020' | head -1
hdfs://sandbox-hdp.hortonworks.com

Then tries to update location with same command as below (except dryRun), which breaks Hive completely.

[hive@sandbox-hdp ~]$ hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -updateLocation hdfs://sandbox-hdp.hortonworks.com:8020 hdfs://sandbox-hdp.hortonworks.com -dryRun
Initializing HiveMetaTool..
... (snip)...
Looking for LOCATION_URI field in DBS table to update..
Dry Run of updateLocation on table DBS..
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db
old location: hdfs://sandbox-hdp.hortonworks.com/apps/hive/warehouse/dummies.db new location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/xademo.db
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse
Found 4 records in DBS table to update
Looking for LOCATION field in SDS table to update..
Dry Run of updateLocation on table SDS..
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=A new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=A
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=B new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=B
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/hdfs_audit new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/hdfs_audit
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/test2 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/test2
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db/recharge_details new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/xademo.db/recharge_details
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/sample_07_orc new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/sample_07_orc
old location: hdfs://sandbox-hdp.hortonworks.com/tmp/emp_stage_data new location: hdfs://sandbox-hdp.hortonworks.com:8020/tmp/emp_stage_data
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=C new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=C
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=D new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=D
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db/call_detail_records new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/xademo.db/call_detail_records
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db/customer_details new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/xademo.db/customer_details
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/customer new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/customer
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/sample_08 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/sample_08
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/product new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/product
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/sample_07 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/sample_07
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/sales_fact_1998 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/sales_fact_1998
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/inventory_fact_1998 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/inventory_fact_1998
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/census new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/census
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/store new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/store
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/census_clus new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/census_clus
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/sample_08 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/sample_08
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/sample_07 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/sample_07
Found 23 records in SDS table to update

Lines from output-9505.txt

2018-02-27 20:16:42,747 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -listFSRoot' 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://farmerm.hw262.com:8020' | head -1'] {}
2018-02-27 20:16:48,897 - call returned (0, 'hdfs://farmerm.hw262.com')
2018-02-27 20:16:48,898 - Execute['hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -updateLocation hdfs://farmerm.hw262.com:8020 hdfs://farmerm.hw262.com'] {'environment': {'PATH': u'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/current/hadoop-client/bin'}, 'user': 'hive'}