Bulk Operation

  1. Choose Issues
  2. Choose Operation
  3. Operation Details
  4. Confirmation

Step 1 of 4: Choose Issues

Cancel

T Patch Info Key Summary Assignee Reporter P Status Resolution Created Updated Due Development
Sub-task HADOOP-17017

HADOOP-15620 S3A client retries on SSL Auth exceptions triggered by "." bucket names

Unassigned Steve Loughran Minor Open Unresolved  
Sub-task HADOOP-16939

HADOOP-15620 fs.s3a.authoritative.path should support multiple FS URIs

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16853

HADOOP-15620 ITestS3GuardOutOfBandOperations failing on versioned S3 buckets

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16827

HADOOP-15620 TestHarFileSystem.testInheritedMethodsImplemented broken

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16823

HADOOP-15620 Large DeleteObject requests are their own Thundering Herd

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16801

HADOOP-15620 S3Guard listFiles will not query S3 if all listings are authoritative

Mustafa Iman Mustafa Iman Minor Resolved Fixed  
Sub-task HADOOP-16794

HADOOP-15620 S3A reverts KMS encryption to the bucket's default KMS key in rename/copy

Mukund Thakur Mukund Thakur Major Resolved Fixed  
Sub-task HADOOP-16792

HADOOP-15620 Let s3 clients configure request timeout

Mustafa Iman Mustafa Iman Major Resolved Fixed  
Sub-task HADOOP-16767

HADOOP-15620 S3AInputStream reopening does not handle non IO exceptions properly

Sergei Poganshev Sergei Poganshev Major Resolved Fixed  
Sub-task HADOOP-16759

HADOOP-15620 Filesystem openFile() builder to take a FileStatus param

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16758

HADOOP-15620 Refine testing.md to tell user better how to use auth-keys.xml

Mingliang Liu Mingliang Liu Minor Resolved Fixed  
Sub-task HADOOP-16757

HADOOP-15620 Increase timeout unit test rule for MetadataStoreTestBase

Mingliang Liu Mingliang Liu Major Resolved Fixed  
Sub-task HADOOP-16751

HADOOP-15620 DurationInfo text parsing/formatting should be moved out of hotpath

Rajesh Balamohan Rajesh Balamohan Minor Resolved Fixed  
Sub-task HADOOP-16746

HADOOP-15620 S3A empty dir markers are not created in s3guard as authoritative

Steve Loughran Steve Loughran Critical Resolved Fixed  
Sub-task HADOOP-16732

HADOOP-15620 S3Guard to support encrypted DynamoDB table

Mingliang Liu Mingliang Liu Major Resolved Fixed  
Sub-task HADOOP-16725

HADOOP-15620 s3guard prune can delete directories -leaving orphan children.

Steve Loughran Steve Loughran Critical Resolved Invalid  
Sub-task HADOOP-16722

HADOOP-15620 S3GuardTool to support FilterFileSystem

Steve Loughran Mingliang Liu Major Resolved Fixed  
Sub-task HADOOP-16711

HADOOP-15620 S3A bucket existence checks to support v2 API and "no checks at all"

Mukund Thakur Rajesh Balamohan Minor Resolved Fixed  
Sub-task HADOOP-16709

HADOOP-15620 S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths

Gabor Bota Rajesh Balamohan Minor Resolved Fixed  
Sub-task HADOOP-16708

HADOOP-15620 HadoopExecutors cleanup to only log at debug

David Mollitor Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16653

HADOOP-15620 S3Guard DDB overreacts to no tag access

Gabor Bota Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16651

HADOOP-15620 S3 getBucketLocation() can return "US" for us-east

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16650

HADOOP-15620 ITestS3AClosedFS failing -junit test thread

Steve Loughran Steve Loughran Blocker Resolved Fixed  
Sub-task HADOOP-16645

HADOOP-15620 S3A Delegation Token extension point to use StoreContext

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16635

HADOOP-15620 S3A innerGetFileStatus s"directories only" scan still does a HEAD

Steve Loughran Steve Loughran Blocker Resolved Fixed  
Sub-task HADOOP-16634

HADOOP-15620 S3A ITest failures without S3Guard

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16632

HADOOP-15620 Speculating & Partitioned S3A magic committers can leave pending files under __magic

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16626

HADOOP-15620 S3A ITestRestrictedReadAccess fails

Steve Loughran Siddharth Seth Major Resolved Fixed  
Sub-task HADOOP-16607

HADOOP-15620 s3a attempts to look up password/encryption fail if JCEKS file unreadable

Unassigned Steve Loughran Minor Resolved Duplicate  
Sub-task HADOOP-16591

HADOOP-15620 S3A ITest*MRjob failures

Siddharth Seth Siddharth Seth Major Resolved Fixed  
Sub-task HADOOP-16570

HADOOP-15620 S3A committers leak threads/raises OOM on job/task commit at scale

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16500

HADOOP-15620 S3ADelegationTokens to only log at debug on startup

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16499

HADOOP-15620 S3A retry policy to be exponential

Steve Loughran Steve Loughran Critical Resolved Fixed  
Sub-task HADOOP-16478

HADOOP-15620 S3Guard bucket-info fails if the bucket location is denied to the caller

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16477

HADOOP-15620 S3A delegation token tests fail if fs.s3a.encryption.key set

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16471

HADOOP-15620 Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider

Steve Loughran Steve Loughran Major Resolved Done  
Sub-task HADOOP-16470

HADOOP-15620 Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16469

HADOOP-15620 Typo in s3a committers.md doc

Unassigned Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16458

HADOOP-15620 LocatedFileStatusFetcher scans failing intermittently against S3 store

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16450

HADOOP-15620 ITestS3ACommitterFactory failing, S3 client is not inconsistent

Steve Loughran Steve Loughran Trivial Resolved Fixed  
Sub-task HADOOP-16427

HADOOP-15620 Downgrade INFO message on rm s3a root dir to DEBUG

Unassigned Steve Loughran Trivial Resolved Won't Fix  
Sub-task HADOOP-16422

HADOOP-15620 AbstractITCommitMRJob.testMRJob test failures

Unassigned Gabor Bota Major Resolved Duplicate  
Sub-task HADOOP-16420

HADOOP-15620 S3A returns 400 "bad request" on a single path within an S3 bucket

Unassigned Steve Loughran Minor Resolved Cannot Reproduce  
Sub-task HADOOP-16387

HADOOP-15620 S3A openFile() options to allow etag/version to be set

Unassigned Steve Loughran Minor Resolved Duplicate  
Sub-task HADOOP-16379

HADOOP-15620 S3AInputStream#unbuffer should merge input stream stats into fs-wide stats

Sahil Takiar Sahil Takiar Major Resolved Fixed  
Sub-task HADOOP-16371

HADOOP-15620 Option to disable GCM for SSL connections when running on Java 8

Sahil Takiar Sahil Takiar Major Resolved Fixed  
Sub-task HADOOP-16360

HADOOP-15620 S3A NullPointerException: null uri host. This can be caused by unencoded / in the password string

Unassigned Ladislav Jech Minor Resolved Won't Fix  
Sub-task HADOOP-16357

HADOOP-15620 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists

Steve Loughran Prabhu Joseph Minor Resolved Fixed  
Sub-task HADOOP-16347

HADOOP-15620 MapReduce job tasks fails on S3A ssl3_get_server_certificate:certificate verify

Steve Loughran Prabhu Joseph Major Resolved Cannot Reproduce  
Sub-task HADOOP-16346

HADOOP-15620 Stabilize S3A OpenSSL support

Sahil Takiar Steve Loughran Blocker Resolved Fixed  
Sub-task HADOOP-16333

HADOOP-15620 Test Hang in S3A S3guard test MetadataStoreTestBase.testListChildren

Unassigned Steve Loughran Major Resolved Cannot Reproduce  
Sub-task HADOOP-16332

HADOOP-15620 Remove S3A's depedency on http core

Steve Loughran Steve Loughran Critical Resolved Fixed  
Sub-task HADOOP-16330

HADOOP-15620 Regression: TestStagingPartitionedJobCommit failing with empty etag list

Steve Loughran Steve Loughran Minor Resolved Done  
Sub-task HADOOP-16328

HADOOP-15620 ClassCastException in S3GuardTool.checkMetadataStoreUri

Steve Loughran Steve Loughran Major Resolved Cannot Reproduce  
Sub-task HADOOP-16327

HADOOP-15620 s3a test docs to mention non-auth; or s3a tests to default to non-auth

Unassigned Steve Loughran Major Resolved Invalid  
Sub-task HADOOP-16324

HADOOP-15620 S3A Delegation Token code to spell "Marshalled" as Marshaled

Steve Loughran Steve Loughran Blocker Resolved Won't Fix  
Sub-task HADOOP-16319

HADOOP-15620 S3A Etag tests fail with default encryption enabled on bucket

Ben Roling Ben Roling Minor Resolved Fixed  
Sub-task HADOOP-16316

HADOOP-15620 S3A delegation tests fail if you set fs.s3a.secret.key

Unassigned Steve Loughran Minor Resolved Duplicate  
Sub-task HADOOP-16233

HADOOP-15620 S3AFileStatus to declare that isEncrypted() is always true

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16207

HADOOP-15620 Improved S3A MR tests

Steve Loughran Steve Loughran Critical Resolved Fixed  
Sub-task HADOOP-16201

HADOOP-15620 S3AFileSystem#innerMkdirs builds needless lists

Lokesh Jain Lokesh Jain Trivial Resolved Fixed  
Sub-task HADOOP-16197

HADOOP-15620 S3AUtils.translateException to map CredentialInitializationException to AccessDeniedException

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16195

HADOOP-15620 S3A MarshalledCredentials.toString() doesn't print full date/time of expiry

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16190

HADOOP-15620 S3A copyFile operation to include source versionID or etag in the copy request

Steve Loughran Steve Loughran Major Resolved Duplicate  
Sub-task HADOOP-16177

HADOOP-15620 regression: ITestS3AMiniYarnCluster failing on branch-3.2

Unassigned Steve Loughran Major Resolved Invalid  
Sub-task HADOOP-16175

HADOOP-15620 S3Guard to add DynamoDBLocal Support

lqjacklee lqjacklee Major Resolved Won't Fix  
Sub-task HADOOP-16124

HADOOP-15620 Extend documentation in testing.md about endpoint constants

Adam Antal Adam Antal Trivial Resolved Fixed  
Sub-task HADOOP-16117

HADOOP-15620 Update AWS SDK to 1.11.563

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16109

HADOOP-15620 Parquet reading S3AFileSystem causes EOF

Steve Loughran Dave Christianson Blocker Resolved Fixed  
Sub-task HADOOP-16093

HADOOP-15620 Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util

Abhishek Modi Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-16079

HADOOP-15620 Token.toString faulting if any token listed can't load.

Steve Loughran Steve Loughran Blocker Resolved Fixed  
Sub-task HADOOP-16058

HADOOP-15620 S3A tests to include Terasort

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-16050

HADOOP-15620 S3A SSL connections should use OpenSSL

Sahil Takiar Justin Uang Major Resolved Fixed  
Sub-task HADOOP-16033

HADOOP-15620 hamcrest-library declaration in hadoop-aws to be scoped test

Steve Loughran Steve Loughran Blocker Resolved Duplicate  
Sub-task HADOOP-16027

HADOOP-15620 [DOC] Effective use of FS instances during S3A integration tests

Gabor Bota Gabor Bota Major Resolved Fixed  
Sub-task HADOOP-16015

HADOOP-15620 Add bouncycastle jars to hadoop-aws as test dependencies

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-15990

HADOOP-15620 S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2

lqjacklee Steve Loughran Major Resolved Duplicate  
Sub-task HADOOP-15961

HADOOP-15620 S3A committers: make sure there's regular progress() calls

lqjacklee Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-15932

HADOOP-15620 Oozie unable to create sharelib in s3a filesystem

Steve Loughran Soumitra Sulav Critical Resolved Fixed  
Sub-task HADOOP-15924

HADOOP-15620 Hadoop aws does not use shaded jars

Unassigned Bharat Viswanadham Major Resolved Won't Fix  
Sub-task HADOOP-15920

HADOOP-15620 get patch for S3a nextReadPos(), through Yetus

lqjacklee Steve Loughran Major Resolved Done  
Sub-task HADOOP-15890

HADOOP-15620 Some S3A committer tests don't match ITest* pattern; don't run in maven

Steve Loughran Steve Loughran Major Resolved Duplicate  
Sub-task HADOOP-15848

HADOOP-15620 ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error

Ewan Higgs Gabor Bota Major Resolved Fixed  
Sub-task HADOOP-15839

HADOOP-15620 Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-15826

HADOOP-15620 @Retries annotation of putObject() call & uses wrong

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-15806

HADOOP-15620 Move ITestS3AMiniYarnCluster to S3A committers

Steve Loughran Steve Loughran Major Resolved Duplicate  
Sub-task HADOOP-15750

HADOOP-15620 remove obsolete S3A test ITestS3ACredentialsInURL

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-15748

HADOOP-15620 S3 listing inconsistency can raise NPE in globber

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-15747

HADOOP-15620 warning about user:pass in URI to explicitly call out Hadoop 3.2 as removal

Steve Loughran Steve Loughran Major Resolved Won't Fix  
Sub-task HADOOP-15729

HADOOP-15620 [s3a] stop treat fs.s3a.max.threads as the long-term minimum

Sean Mackrory Sean Mackrory Major Resolved Fixed  
Sub-task HADOOP-15672

HADOOP-15620 add s3guard CLI command to generate session keys for an assumed role

Steve Loughran Steve Loughran Minor Resolved Duplicate  
Sub-task HADOOP-15667

HADOOP-15620 FileSystemMultipartUploader should verify that UploadHandle has non-0 length

Ewan Higgs Ewan Higgs Major Resolved Fixed  
Sub-task HADOOP-15658

HADOOP-15620 Memory leak in S3AOutputStream

Steve Loughran Piotr Nowojski Major Resolved Fixed  
Sub-task HADOOP-15625

HADOOP-15620 S3A input stream to use etags/version number to detect changed source files

Ben Roling Brahma Reddy Battula Major Resolved Fixed  
Sub-task HADOOP-15604

HADOOP-15620 Bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard

Steve Loughran Gabor Bota Major Resolved Fixed  
Sub-task HADOOP-15421

HADOOP-15620 Stabilise/formalise the JSON _SUCCESS format used in the S3A committers

Unassigned Steve Loughran Major Resolved Won't Fix  
Sub-task HADOOP-15409

HADOOP-15620 S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2

lqjacklee Steve Loughran Minor Resolved Duplicate  
Sub-task HADOOP-15370

HADOOP-15620 S3A log message on rm s3a://bucket/ not intuitive

Gabor Bota Steve Loughran Trivial Resolved Fixed  
Sub-task HADOOP-15364

HADOOP-15620 Add support for S3 Select to S3A

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-15228

HADOOP-15620 S3A Retry policy to retry on NoResponseException

Steve Loughran Steve Loughran Minor Resolved Duplicate  
Sub-task HADOOP-15135

HADOOP-15620 S3a to support get/set permissions through S3 object tags

Unassigned Steve Loughran Major Resolved Won't Fix  
Sub-task HADOOP-15091

HADOOP-15620 S3aUtils.getEncryptionAlgorithm() always logs@Debug "Using SSE-C"

Unassigned Steve Loughran Trivial Resolved Duplicate  
Sub-task HADOOP-14762

HADOOP-15620 S3A warning of obsolete encryption key which is never used

Unassigned Steve Loughran Minor Resolved Duplicate  
Sub-task HADOOP-14747

HADOOP-15620 S3AInputStream to implement CanUnbuffer

Sahil Takiar Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-14621

HADOOP-15620 S3A client raising ConnectionPoolTimeoutException in test

Unassigned Steve Loughran Minor Resolved Cannot Reproduce  
Sub-task HADOOP-14556

HADOOP-15620 S3A to support Delegation Tokens

Steve Loughran Steve Loughran Major Resolved Fixed  
Sub-task HADOOP-14551

HADOOP-15620 S3A init hangs if you try to connect while the system is offline

Unassigned Steve Loughran Minor Resolved Won't Fix  
Sub-task HADOOP-14124

HADOOP-15620 S3AFileSystem silently deletes "fake" directories when writing a file.

Unassigned Joel Baranick Minor Resolved Won't Fix  
Sub-task HADOOP-13884

HADOOP-15620 s3a create(overwrite=true) to only look for dir/ and list entries, not file

Steve Loughran Steve Loughran Minor Resolved Duplicate  
Sub-task HADOOP-13656

HADOOP-15620 fs -expunge to take a filesystem

Shweta Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-13600

HADOOP-15620 S3a rename() to copy files in a directory in parallel

Unassigned Steve Loughran Major Resolved Duplicate  
Sub-task HADOOP-13432

HADOOP-15620 S3A: Consider using TransferManager.download for copyToLocalFile

Unassigned Rajesh Balamohan Minor Resolved Won't Fix  
Sub-task HADOOP-13402

HADOOP-15620 S3A should allow renaming to a pre-existing destination directory to move the source path under that directory, similar to HDFS.

Unassigned Rajesh Balamohan Minor Resolved Won't Fix  
Sub-task HADOOP-13373

HADOOP-15620 Add S3A implementation of FSMainOperationsBaseTest

Steve Loughran Steve Loughran Minor Resolved Fixed  
Sub-task HADOOP-13330

HADOOP-15620 Parallelize S3A directory deletes

Unassigned Steve Loughran Major Resolved Duplicate  
Sub-task HADOOP-12949

HADOOP-15620 Add HTrace to the s3a connector

Madhawa Kasun Gunasekara Madhawa Gunasekara Major Resolved Won't Fix  

Cancel