Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15797

optional / builtin modules confused for cloud storage

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 3.2.0, 3.1.1
    • None
    • fs/adl, fs/azure, fs/s3
    • None

    Description

      Throwing this in your .hadooprc results in hadoop-aws being in the classpath but not hadoop-azure*:

      hadoop_add_to_classpath_tools hadoop-aws
      hadoop_add_to_classpath_tools hadoop-azure
      hadoop_add_to_classpath_tools hadoop-azure-datalake

      It would seem that the core issue is that that requires the module to have listed it's dependencies in MODULE_NAME.tools-builtin.txt, whereas the Azure connectors only have them listed in MODULE_NAME.tools-optional.txt. S3 does both, and there's a comment in it's POM about how it needs to do this because of the "hadoop s3guard" CLI.

      Maybe there's some history that I'm missing here, but I think what's wrong here is that hadoop_add_to_classpath should get what it needs from optional modules. builtin modules shouldn't even need hadoop_add_to_classpath to be added anyway.

      Attachments

        Activity

          People

            Unassigned Unassigned
            mackrorysd Sean Mackrory
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: