Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15797

optional / builtin modules confused for cloud storage

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 3.2.0, 3.1.1
    • Fix Version/s: None
    • Component/s: fs/adl, fs/azure, fs/s3
    • Labels:
      None

      Description

      Throwing this in your .hadooprc results in hadoop-aws being in the classpath but not hadoop-azure*:

      hadoop_add_to_classpath_tools hadoop-aws
      hadoop_add_to_classpath_tools hadoop-azure
      hadoop_add_to_classpath_tools hadoop-azure-datalake

      It would seem that the core issue is that that requires the module to have listed it's dependencies in MODULE_NAME.tools-builtin.txt, whereas the Azure connectors only have them listed in MODULE_NAME.tools-optional.txt. S3 does both, and there's a comment in it's POM about how it needs to do this because of the "hadoop s3guard" CLI.

      Maybe there's some history that I'm missing here, but I think what's wrong here is that hadoop_add_to_classpath should get what it needs from optional modules. builtin modules shouldn't even need hadoop_add_to_classpath to be added anyway.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mackrorysd Sean Mackrory
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: