Hadoop Common
  1. Hadoop Common
  2. HADOOP-9385

create hadoop-common-project/hadoop-filesystem-clients subprojects for blobstore & other clients

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      As discussed on hadoop-general, we need somewhere to host the non-HDFS filesystem clients. S3/S3N, ftp are all in hadoop-common, with the JAR dependencies there. This doesn't scale to openstack, azure, or handle changes in the S3 dependencies.

      With a project of hadoop-common/hadoop-filesystem-clients, we could add separate FS clients: hadoop-filesystem-client-aws, hadoop-filesystem-client-openstack, etc, each with their own tests, JARs and POM file dependencies. This would translate into separate bigtop RPMs/JARs

        Issue Links

          Activity

          Hide
          Steve Loughran added a comment -

          Once a dir is chosen we can create the toplevel POM, then add the HADOOP-8545 Swift client, and then move S3/S3n support when adopting the new S3 client library - HADOOP-9384 -

          Each client should also include the new filesystem tests of HADOOP-9361, skipping them all if the blobstore credentials aren't supplied in its test/resources/config.xml file

          Show
          Steve Loughran added a comment - Once a dir is chosen we can create the toplevel POM, then add the HADOOP-8545 Swift client, and then move S3/S3n support when adopting the new S3 client library - HADOOP-9384 - Each client should also include the new filesystem tests of HADOOP-9361 , skipping them all if the blobstore credentials aren't supplied in its test/resources/config.xml file
          Hide
          Alejandro Abdelnur added a comment -

          Along the lines of what I've just commented on the email thread,

          Unless those filesystems are part of hadoop, their clients should not be distributed/build by hadoop. My concern is testability and maintainability.

          Show
          Alejandro Abdelnur added a comment - Along the lines of what I've just commented on the email thread, Unless those filesystems are part of hadoop, their clients should not be distributed/build by hadoop. My concern is testability and maintainability.
          Hide
          Luke Lu added a comment -

          +1 for moving all existing external fs client support to separate jars. IMO, "hadoop-fs-extra" probably conveys the purpose of the project better: it's for external/extra fs support

          Though I share the maintainability concerns, there is substantial end-user benefit of keeping the hadoop client code for these fs together. We can put a README in the project with standard disclaimers...

          Show
          Luke Lu added a comment - +1 for moving all existing external fs client support to separate jars. IMO, "hadoop-fs-extra" probably conveys the purpose of the project better: it's for external/extra fs support Though I share the maintainability concerns, there is substantial end-user benefit of keeping the hadoop client code for these fs together. We can put a README in the project with standard disclaimers...
          Hide
          Steve Loughran added a comment -

          Alejandro Abdelnur -I agree with you for the third-party filesystems -that creates a set of classes that are untestable without access to the specific cluster type, and so creates unsupportable code.

          Blobstores are more tractable as they are universally accessible, given sufficient funding.

          Show
          Steve Loughran added a comment - Alejandro Abdelnur -I agree with you for the third-party filesystems -that creates a set of classes that are untestable without access to the specific cluster type, and so creates unsupportable code. Blobstores are more tractable as they are universally accessible, given sufficient funding.

            People

            • Assignee:
              Unassigned
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development