[HADOOP-16408] `hadoop classpath` does not export azure jars, only data lake

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: 3.2.0
• Component/s: fs/azure
• Assignee: Unassigned
• Reporter: Jose Luis Pedrosa
• Created: 03/Jul/19 11:59
• Updated: 05/Jul/19 10:06
• Resolved: 05/Jul/19 10:06

Description

`hadoop classpath` does not export azure jars, only azure data lake jars

This means that when running spark applications, it will not be able to use abfs urls (among other things).

When running:

hadoop-3.2.0/bin/hadoop classpath

It will output all the jars for hadoop-azure-datalake, but will not output the jars for hadoop-azure, which are the ones that actually contain the FS for the new ABFS. 

/hadoop-3.2.0/etc/hadoop:
/hadoop-3.2.0/share/hadoop/common/lib/*:
/hadoop-3.2.0/share/hadoop/common/*:
/hadoop-3.2.0/share/hadoop/tools/lib/aliyun-sdk-oss-2.8.3.jar:
/hadoop-3.2.0/share/hadoop/tools/lib/jdom-1.1.jar:
/hadoop-3.2.0/share/hadoop/tools/lib/hadoop-aliyun-3.2.0.jar:
/hadoop-3.2.0/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar:
/hadoop-3.2.0/share/hadoop/tools/lib/hadoop-aws-3.2.0.jar:
/hadoop-3.2.0/share/hadoop/tools/lib/azure-data-lake-store-sdk-2.2.9.jar:
/hadoop-3.2.0/share/hadoop/tools/lib/hadoop-azure-datalake-3.2.0.jar:
/hadoop-3.2.0/share/hadoop/hdfs:
/hadoop-3.2.0/share/hadoop/hdfs/lib/*:
/hadoop-3.2.0/share/hadoop/hdfs/*:
/hadoop-3.2.0/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar:
/hadoop-3.2.0/share/hadoop/tools/lib/lz4-1.2.0.jar:
/hadoop-3.2.0/share/hadoop/tools/lib/hadoop-kafka-3.2.0.jar:
/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:
/hadoop-3.2.0/share/hadoop/mapreduce/*:
/hadoop-3.2.0/share/hadoop/tools/lib/hadoop-openstack-3.2.0.jar:
/hadoop-3.2.0/share/hadoop/yarn:
/hadoop-3.2.0/share/hadoop/yarn/lib/*:
/hadoop-3.2.0/share/hadoop/yarn/*