[HADOOP-17547] Magic committer to downgrade abort in cleanup if list uploads fails with access denied - ASF JIRA

Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.3.0
Fix Version/s: 3.3.2
Component/s: fs/s3
Labels:
- pull-request-available

Description

If the caller doesn't have "s3:ListBucketMultipartUploads" permissions on a bucket, then magic committer cleanup fails.

	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:247)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:112)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:315)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:311)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:286)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.listMultipartUploads(S3AFileSystem.java:4549)
	at org.apache.hadoop.fs.s3a.commit.CommitOperations.listPendingUploadsUnderPath(CommitOperations.java:361)
	at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.abortPendingUploadsInCleanup(AbstractS3ACommitter.java:671)
	at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.cleanup(AbstractS3ACommitter.java:770)

it should just swallow this, given it's best effort

Attachments

Issue Links

Add Link

links to

GitHub Pull Request #3051

Delete this link

Activity

Comment

This comment will be Viewable by All Users Viewable by All Users

Cancel

People

Assignee:: Bogdan Stolojan

Reporter:: Steve Loughran

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 25/Feb/21 11:21

Updated:: 14/Jun/21 07:57

Resolved:: 12/Jun/21 16:47

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

Magic committer to downgrade abort in cleanup if list uploads fails with access denied