[HADOOP-13377] AliyunOSS: improvements for stabilization and optimization

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha2
• Component/s: fs, fs/oss
• Target Version/s: 3.4.0
• Assignee: Genmao Yu
• Reporter: Genmao Yu
• Created: 14/Jul/16 08:00
• Updated: 09/Apr/20 18:52

Description

This work is to handle some stabilization issues.

Issue Links

• relates to: HADOOP-14649 Update aliyun-sdk-oss version to 2.8.1
• relates to: HADOOP-14964 AliyunOSS: backport Aliyun OSS module to branch-2

Sub-Tasks

• Aliyun Connection broken when idle then 1 minutes or build than 3 hours: Resolved
• AliyunOSS: handle the failure in the batch delete operation `deleteDirs`.: Resolved
• AliyunOSS: Failed to read from stream when seek beyond the download size: Resolved
• AliyunOSS: update oss sdk version: Resolved
• AliyunOSS: Unit test failed in TestAliyunOSSContractCreate: Resolved
• AliyunOSS: oss directory filestatus should use meta time: Resolved
• AliyunOSS: listStatus returns wrong file info: Resolved
• AliyunOSS: improve the performance of object metadata operation: In Progress
• Aliyun OSS FileSystem contract test should implement getTestBaseDir(): Resolved
• Aliyun OSS should not use empty endpoint as default: Resolved
• AliyunOSS: Implement the `createNonRecursive` operator: Resolved
• AliyunOSS: Override listFiles and listLocatedStatus: Resolved
• AliyunOSS：Improve oss error logging: Resolved
• Add hadoop-aliyun as dependency of hadoop-cloud-storage: Resolved
• AliyunOSS: provide one asynchronous multi-part based uploading mechanism: Resolved
• AliyunOSS: provide one memory-based buffering mechanism in outpustream to oss: Open
• AliyunOSS: refactor storage statistics to oss: Open
• AliyunOSS: support user agent configuration and include that & Hadoop version information to oss server: Resolved
• AliyunOSS: Add User-Agent header in HTTP requests to the OSS server: Resolved
• AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance: Resolved
• IOException may be thrown when read from Aliyun OSS in some case: Resolved
• AliyunOSS: move files under a directory in parallel when rename a directory: Resolved
• AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore: Resolved
• AliyunOSS: Supports role based credential in URL: Resolved
• AliyunOSS: Support Assume Roles in AliyunOSS: Resolved
• AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy: Resolved
• AliyunOSS: update oss-sdk version to 3.0.0: Resolved
• AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream: Resolved
• AliyunOSS: fix incorrect ReadOps and WriteOps in statistics: Resolved
• AliyunOSS: improvements for copyFile/copyDirectory and logging: Resolved
• AliyunOSS: Remove temporary files when upload small files to OSS: Resolved
• AliyunOSS: support ListObjectsV2: Resolved