[HDFS-6200] Create a separate jar for hdfs-client

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: build
• Target Version/s: 2.8.0

Release Note

Projects that access HDFS can depend on the hadoop-hdfs-client module instead of the hadoop-hdfs module to avoid pulling in unnecessary dependency.
Please note that hadoop-hdfs-client module could miss class like ConfiguredFailoverProxyProvider. So if a cluster is in HA deployment, we should still use hadoop-hdfs instead.

• Assignee: Haohui Mai
• Reporter: Haohui Mai
• Created: 07/Apr/14 23:25
• Updated: 16/Mar/17 18:15
• Resolved: 28/Oct/15 23:22

Description

Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs client. As discussed in the hdfs-dev mailing list (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), downstream projects are forced to bring in additional dependency in order to access hdfs. The additional dependency sometimes can be difficult to manage for projects like Apache Falcon and Apache Oozie.

This jira proposes to create a new project, hadoop-hdfs-cliient, which contains the client side of the hdfs code. Downstream projects can use this jar instead of the hadoop-hdfs to avoid unnecessary dependency.

Note that it does not break the compatibility of downstream projects. This is because old downstream projects implicitly depend on hadoop-hdfs-client through the hadoop-hdfs jar.

Issue Links

• breaks: HDFS-9301 HDFS clients can't construct HdfsConfiguration instances
• is depended upon by: HDFS-6566 Separate public and private keys in DFSConfigKey
• is depended upon by: HDFS-6564 Use slf4j instead of common-logging in hdfs-client
• is depended upon by: HDFS-6567 Normalize the order of public final in HdfsFileStatus
• is depended upon by: HDFS-6565 Use jackson instead jetty json in hdfs-client
• is related to: HADOOP-11656 Classpath isolation for downstream clients
• relates to: HADOOP-1815 Separate client and server jars
• relates to: HDFS-9815 Move o.a.h.fs.Hdfs to hadoop-hdfs-client
• relates to: HADOOP-10741 A lightweight WebHDFS client library
• relates to: HADOOP-12499 Create a client jar for hadoop-common
• relates to: HDFS-8048 Umbrella jira for moving HDFS client implmentation to a package

Sub-Tasks

• Update the POM to create a separate hdfs-client jar: Resolved
• Move WebHdfsFileSystem into hadoop-hdfs-client: Resolved
• Move DFSIn/OutputStream and related classes to hadoop-hdfs-client: Resolved
• Move DFSInputStream and related classes to hadoop-hdfs-client: Resolved
• Separate JSON related routines used by WebHdfsFileSystem to a package local class: Resolved
• Move o.a.h.hdfs.web.resources.* to the client jars: Resolved
• Move BlockTokenSecretManager.AccessMode into BlockTokenIdentifier: Resolved
• hadoop-hdfs-client dependency convergence error: Resolved
• Move GRANDFATHER_GENERATION_STAMP and GRANDFATER_INODE_ID to hdfs-client: Resolved
• Move LocatedBlocks and related classes to hdfs-client: Resolved
• Separate client related routines in HAUtil into a new class: Resolved
• Move classes that used by ClientProtocol into hdfs-client: Resolved
• Move all protocol classes used by ClientProtocol to hdfs-client: Resolved
• Move ClientProtocol to the hdfs-client: Resolved
• Move HdfsServerConstants#IO_FILE_BUFFER_SIZE and SMALL_BUFFER_SIZE to the users: Resolved
• Clean up the checkstyle warinings about ClientProtocol: Resolved
• Move protobuf files that define the client-sever protocols to hdfs-client: Resolved
• Clean up the import statements in ClientProtocol: Resolved
• Keep server related definition in hdfs.proto on server side: Resolved
• Move DistributedFileSystem to hadoop-hdfs-client: Resolved
• Move DfsClientConf to hdfs-client: Resolved
• Move BlockReaderLocal to hdfs-client: Resolved
• Move ShortCircuitShm to hdfs-client: Resolved
• Use GenericTestUtils to set log levels in TestPread and TestReplaceDatanodeOnFailure: Resolved
• Move the shortcircuit package to hdfs-client: Resolved
• Clean up checkstyle warnings in o.a.h.hdfs.DfsClientConf: Resolved
• Clean up checkstyle warnings in shortcircuit package: Resolved
• Remove guards when calling LOG.debug() and LOG.trace() in client package: Resolved
• Move RemoteBlockReader to hdfs-client module: Resolved
• Move o.a.h.hdfs.net/*Peer classes to hdfs-client: Resolved
• Replace NameNode.DEFAULT_PORT with HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT config key: Resolved
• Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to hadoop-hdfs-client module: Resolved
• Update the pom to add junit dependency and move TestXAttr to client project: Resolved
• Move NameNode.getAddress() and NameNode.getUri() to hadoop-hdfs-client: Resolved
• Separate client and server side methods of o.a.h.hdfs.NameNodeProxies: Resolved
• Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client: Resolved
• Move DistributedFileSystem to hadoop-hdfs-client: Resolved
• Remove deprecated NameNode.getUri() static helper method: Resolved
• Move hdfs-client protobuf convert methods from PBHelper to PBHelperClient: Resolved
• Move config keys used by hdfs-client to HdfsClientConfigKeys: Resolved
• Move LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD constants from HdfsServerConstants to HdfsConstants: Resolved
• Move DFSClient to hadoop-hdfs-client: Resolved
• Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client: Resolved
• Move hftp / hsftp filesystem to hfds-client: Resolved
• Update pom.xml in other modules to depend on hdfs-client instead of hdfs: Resolved
• Move client side unit test to hadoop-hdfs-client: Resolved
• Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client: Resolved
• Add HdfsClientConfigKeys class to TestHdfsConfigFields#configurationClasses: Resolved