Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16408

`hadoop classpath` does not export azure jars, only data lake

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Invalid
    • Affects Version/s: 3.2.0
    • Fix Version/s: None
    • Component/s: fs/azure
    • Labels:
      None

      Description

      `hadoop classpath` does not export azure jars, only azure data lake jars

      This means that when running spark applications, it will not be able to use abfs urls (among other things).

      When running:

      hadoop-3.2.0/bin/hadoop classpath
      

      It will output all the jars for hadoop-azure-datalake, but will not output the jars for hadoop-azure, which are the ones that actually contain the FS for the new ABFS. 

      /hadoop-3.2.0/etc/hadoop:
      /hadoop-3.2.0/share/hadoop/common/lib/*:
      /hadoop-3.2.0/share/hadoop/common/*:
      /hadoop-3.2.0/share/hadoop/tools/lib/aliyun-sdk-oss-2.8.3.jar:
      /hadoop-3.2.0/share/hadoop/tools/lib/jdom-1.1.jar:
      /hadoop-3.2.0/share/hadoop/tools/lib/hadoop-aliyun-3.2.0.jar:
      /hadoop-3.2.0/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar:
      /hadoop-3.2.0/share/hadoop/tools/lib/hadoop-aws-3.2.0.jar:
      /hadoop-3.2.0/share/hadoop/tools/lib/azure-data-lake-store-sdk-2.2.9.jar:
      /hadoop-3.2.0/share/hadoop/tools/lib/hadoop-azure-datalake-3.2.0.jar:
      /hadoop-3.2.0/share/hadoop/hdfs:
      /hadoop-3.2.0/share/hadoop/hdfs/lib/*:
      /hadoop-3.2.0/share/hadoop/hdfs/*:
      /hadoop-3.2.0/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar:
      /hadoop-3.2.0/share/hadoop/tools/lib/lz4-1.2.0.jar:
      /hadoop-3.2.0/share/hadoop/tools/lib/hadoop-kafka-3.2.0.jar:
      /hadoop-3.2.0/share/hadoop/mapreduce/lib/*:
      /hadoop-3.2.0/share/hadoop/mapreduce/*:
      /hadoop-3.2.0/share/hadoop/tools/lib/hadoop-openstack-3.2.0.jar:
      /hadoop-3.2.0/share/hadoop/yarn:
      /hadoop-3.2.0/share/hadoop/yarn/lib/*:
      /hadoop-3.2.0/share/hadoop/yarn/*
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jlpedrosa Jose Luis Pedrosa
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: