[HADOOP-12756] Incorporate Aliyun OSS file system implementation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HADOOP-12756
• Fix Version/s: HADOOP-12756, 3.0.0-alpha2, 2.9.1
• Component/s: fs, fs/oss
• Hadoop Flags: Reviewed

Release Note

Aliyun OSS is widely used among China’s cloud users and this work implemented a new Hadoop compatible filesystem AliyunOSSFileSystem with oss scheme, similar to the s3a and azure support.

• Assignee: mingfei.shi
• Reporter: Mingfei Shi
• Created: 02/Feb/16 02:10
• Updated: 03/Sep/19 17:37
• Resolved: 04/Aug/16 13:28

Description

Aliyun OSS is widely used among China’s cloud users, but currently it is not easy to access data laid on OSS storage from user’s Hadoop/Spark application, because of no original support for OSS in Hadoop.

This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, Spark/Hadoop applications can read/write data from OSS without any code change. Narrowing the gap between user’s APP and data storage, like what have been done for S3 in Hadoop

Issue Links

• is related to: HADOOP-16542 Update commons-beanutils version to 1.9.4
• relates to: HADOOP-14649 Update aliyun-sdk-oss version to 2.8.1
• relates to: HADOOP-14964 AliyunOSS: backport Aliyun OSS module to branch-2

Sub-Tasks

• User end documents for Aliyun OSS FileSystem: Resolved
• fix several warnings from findbugs: Resolved
• file-create should throw error rather than overwrite directories: Resolved
• Provide hadoop-aliyun oss configuration documents: Resolved
• the number of multi-part upload part should not bigger than 10000: Resolved
• Support session credentials for authenticating with Aliyun: Resolved
• Do some code refactoring: Resolved
• Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir: Resolved
• Refine credential provider related codes for AliyunOss integration: Resolved
• Clean up AliyunOss integration tests: Resolved
• Rename TestAliyunOSSContractDispCp: Resolved
• Some configuration in Aliyun doc has been outdated: Resolved
• AbstractContractRootDirectoryTest can fail when handling delete "/": Resolved