[HADOOP-16955] Umbrella Jira for improving the Hadoop-cos support in Hadoop

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: fs/cos
• Target Version/s: 3.3.9

Release Note

bugfix:

- resolve the dependency conflict;
- fix the upload buffer returning failed when some exceptions occur;
- fix the issue that the single file upload can not be retried;
- fix the bug of checking if a file exists through listing the file frequently.

features:
- support SessionCredentialsProvider and InstanceCredentialsProvider, which
- allows users to specify the credentials in URI or get it from the CVM (Tencent Cloud Virtual Machine) bound to the CAM role that can access the COS bucket;
- support the server encryption based on SSE-COS and SSE-C;
- support the HTTP proxy settings;
- support the storage class settings;

• Assignee: Yang Yu
• Reporter: Yang Yu
• Due: 06/Apr/20
• Created: 02/Apr/20 17:43
• Updated: 29/Nov/22 15:55

Description

This Umbrella Jira focus on fixing some known bugs and adding some important features.

 

bugfix:

1. resolve the dependency conflict;
2. fix the upload buffer returning failed when some exceptions occur;
3. fix the issue that the single file upload can not be retried;
4. fix the bug of checking if a file exists through listing the file frequently.

features:

1. support SessionCredentialsProvider and InstanceCredentialsProvider, which allows users to specify the credentials in URI or get it from the CVM (Tencent Cloud Virtual Machine) bound to the CAM role that can access the COS bucket;
2. support the server encryption  based on SSE-COS and SSE-C;
3. support the HTTP proxy settings;
4. support the storage class settings;
5. support the CRC64 checksum.

Issue Links

• is related to: HADOOP-15616 Incorporate Tencent Cloud COS File System Implementation

Sub-Tasks

• Resolve hadoop-cos dependency conflict: Resolved
• Optimizing the upload buffer in Hadoop-cos: Patch Available
• Optimizing the retry mechanism when a server error occurs: In Progress
• Supporting the new credentials provider in Hadoop-cos: In Progress
• Supporting the server encryption based on SSE-COS and SSE-C: Open
• Support the CRC64 checksum: Open
• Fix the CosCrendentials Provider in core-site.xml for unit tests.: In Progress
• Improve and revise the performance description in the Tencent COS website document: Patch Available