Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6200

Create a separate jar for hdfs-client

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      Projects that access HDFS can depend on the hadoop-hdfs-client module instead of the hadoop-hdfs module to avoid pulling in unnecessary dependency.
      Please note that hadoop-hdfs-client module could miss class like ConfiguredFailoverProxyProvider. So if a cluster is in HA deployment, we should still use hadoop-hdfs instead.
      Show
      Projects that access HDFS can depend on the hadoop-hdfs-client module instead of the hadoop-hdfs module to avoid pulling in unnecessary dependency. Please note that hadoop-hdfs-client module could miss class like ConfiguredFailoverProxyProvider. So if a cluster is in HA deployment, we should still use hadoop-hdfs instead.

      Description

      Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs client. As discussed in the hdfs-dev mailing list (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), downstream projects are forced to bring in additional dependency in order to access hdfs. The additional dependency sometimes can be difficult to manage for projects like Apache Falcon and Apache Oozie.

      This jira proposes to create a new project, hadoop-hdfs-cliient, which contains the client side of the hdfs code. Downstream projects can use this jar instead of the hadoop-hdfs to avoid unnecessary dependency.

      Note that it does not break the compatibility of downstream projects. This is because old downstream projects implicitly depend on hadoop-hdfs-client through the hadoop-hdfs jar.

        Attachments

        1. HDFS-6200.000.patch
          453 kB
          Haohui Mai
        2. HDFS-6200.001.patch
          478 kB
          Haohui Mai
        3. HDFS-6200.002.patch
          480 kB
          Haohui Mai
        4. HDFS-6200.003.patch
          481 kB
          Haohui Mai
        5. HDFS-6200.004.patch
          481 kB
          Haohui Mai
        6. HDFS-6200.005.patch
          481 kB
          Haohui Mai
        7. HDFS-6200.006.patch
          529 kB
          Haohui Mai
        8. HDFS-6200.007.patch
          537 kB
          Haohui Mai

          Issue Links

          1.
          Update the POM to create a separate hdfs-client jar Sub-task Resolved Haohui Mai
          2.
          Move WebHdfsFileSystem into hadoop-hdfs-client Sub-task Resolved Haohui Mai
          3.
          Move DFSIn/OutputStream and related classes to hadoop-hdfs-client Sub-task Resolved Mingliang Liu
          4.
          Move DFSInputStream and related classes to hadoop-hdfs-client Sub-task Resolved Haohui Mai
          5.
          Separate JSON related routines used by WebHdfsFileSystem to a package local class Sub-task Resolved Haohui Mai
          6.
          Move o.a.h.hdfs.web.resources.* to the client jars Sub-task Resolved Haohui Mai
          7.
          Move BlockTokenSecretManager.AccessMode into BlockTokenIdentifier Sub-task Resolved Haohui Mai
          8.
          hadoop-hdfs-client dependency convergence error Sub-task Resolved Haohui Mai
          9.
          Move GRANDFATHER_GENERATION_STAMP and GRANDFATER_INODE_ID to hdfs-client Sub-task Resolved Haohui Mai
          10.
          Move LocatedBlocks and related classes to hdfs-client Sub-task Resolved Haohui Mai
          11.
          Separate client related routines in HAUtil into a new class Sub-task Resolved Haohui Mai
          12.
          Move classes that used by ClientProtocol into hdfs-client Sub-task Resolved Haohui Mai
          13.
          Move all protocol classes used by ClientProtocol to hdfs-client Sub-task Resolved Haohui Mai
          14.
          Move ClientProtocol to the hdfs-client Sub-task Resolved Takanobu Asanuma
          15.
          Move HdfsServerConstants#IO_FILE_BUFFER_SIZE and SMALL_BUFFER_SIZE to the users Sub-task Resolved Li Lu
          16.
          Clean up the checkstyle warinings about ClientProtocol Sub-task Resolved Takanobu Asanuma
          17.
          Move protobuf files that define the client-sever protocols to hdfs-client Sub-task Resolved Haohui Mai
          18.
          Clean up the import statements in ClientProtocol Sub-task Resolved Takanobu Asanuma
          19.
          Keep server related definition in hdfs.proto on server side Sub-task Resolved Mingliang Liu
          20.
          Move DistributedFileSystem to hadoop-hdfs-client Sub-task Resolved Mingliang Liu
          21.
          Move DfsClientConf to hdfs-client Sub-task Resolved Mingliang Liu
          22.
          Move BlockReaderLocal to hdfs-client Sub-task Resolved Mingliang Liu
          23.
          Move ShortCircuitShm to hdfs-client Sub-task Resolved Mingliang Liu
          24.
          Use GenericTestUtils to set log levels in TestPread and TestReplaceDatanodeOnFailure Sub-task Resolved Mingliang Liu
          25.
          Move the shortcircuit package to hdfs-client Sub-task Resolved Mingliang Liu
          26.
          Clean up checkstyle warnings in o.a.h.hdfs.DfsClientConf Sub-task Resolved Mingliang Liu
          27.
          Clean up checkstyle warnings in shortcircuit package Sub-task Resolved Mingliang Liu
          28.
          Remove guards when calling LOG.debug() and LOG.trace() in client package Sub-task Resolved Mingliang Liu
          29.
          Move RemoteBlockReader to hdfs-client module Sub-task Resolved Mingliang Liu
          30.
          Move o.a.h.hdfs.net/*Peer classes to hdfs-client Sub-task Resolved Mingliang Liu
          31.
          Replace NameNode.DEFAULT_PORT with HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT config key Sub-task Resolved Mingliang Liu
          32.
          Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to hadoop-hdfs-client module Sub-task Resolved Mingliang Liu
          33.
          Update the pom to add junit dependency and move TestXAttr to client project Sub-task Resolved Kanaka Kumar Avvaru
          34.
          Move NameNode.getAddress() and NameNode.getUri() to hadoop-hdfs-client Sub-task Resolved Mingliang Liu
          35.
          Separate client and server side methods of o.a.h.hdfs.NameNodeProxies Sub-task Resolved Mingliang Liu
          36.
          Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client Sub-task Resolved Mingliang Liu
          37.
          Move DistributedFileSystem to hadoop-hdfs-client Sub-task Resolved Mingliang Liu
          38.
          Remove deprecated NameNode.getUri() static helper method Sub-task Resolved Mingliang Liu
          39.
          Move hdfs-client protobuf convert methods from PBHelper to PBHelperClient Sub-task Resolved Mingliang Liu
          40.
          Move config keys used by hdfs-client to HdfsClientConfigKeys Sub-task Resolved Mingliang Liu
          41.
          Move LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD constants from HdfsServerConstants to HdfsConstants Sub-task Resolved Mingliang Liu
          42.
          Move DFSClient to hadoop-hdfs-client Sub-task Resolved Brahma Reddy Battula
          43.
          Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client Sub-task Resolved Mingliang Liu
          44.
          Move hftp / hsftp filesystem to hfds-client Sub-task Resolved Mingliang Liu
          45.
          Update pom.xml in other modules to depend on hdfs-client instead of hdfs Sub-task Resolved Mingliang Liu
          46.
          Move client side unit test to hadoop-hdfs-client Sub-task Resolved Haohui Mai
          47.
          Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client Sub-task Resolved Haohui Mai
          48.
          Add HdfsClientConfigKeys class to TestHdfsConfigFields#configurationClasses Sub-task Resolved Mingliang Liu

            Activity

              People

              • Assignee:
                wheat9 Haohui Mai
                Reporter:
                wheat9 Haohui Mai
              • Votes:
                0 Vote for this issue
                Watchers:
                32 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: