1.
|
make sure staging committers collect DTs for the staging FS |
|
Resolved |
Unassigned
|
|
2.
|
s3a test can hang in teardown with network problems |
|
Resolved |
Unassigned
|
|
3.
|
distcp -update to S3A; abfs, etc always overwrites due to block size mismatch |
|
Resolved |
Steve Loughran
|
|
4.
|
S3AFileSystem.getContentSummary() to use listFiles(recursive) |
|
Resolved |
Unassigned
|
|
5.
|
add extra S3A MPU test to see what happens if a file is created during the MPU |
|
Resolved |
Steve Loughran
|
|
6.
|
support git-secrets commit hook to keep AWS secrets out of git |
|
Patch Available |
Steve Loughran
|
|
7.
|
S3A Secret access to fall back to XML if credential provider raises IOE. |
|
Open |
Unassigned
|
|
8.
|
S3A doesn't actually verify paths have the correct authority |
|
Resolved |
Steve Loughran
|
|
9.
|
Use error code detail in AWS server responses for finer grained exceptions |
|
Resolved |
Unassigned
|
|
10.
|
Test MR split optimisation with recursive listing |
|
Open |
Unassigned
|
|
11.
|
initial part uploads seem to block unnecessarily in S3ABlockOutputStream |
|
Resolved |
Steven Rand
|
|
12.
|
Speed up S3A test runs |
|
Open |
Unassigned
|
|
13.
|
S3AFilesystem.initiateRename() can skip check on dest.parent status if src has same parent |
|
Resolved |
Steve Loughran
|
|
14.
|
S3A copy/rename of large files to be parallelized as a multipart operation |
|
Open |
Unassigned
|
|
15.
|
Test hadoop fs shell against s3a; fix problems |
|
Open |
Unassigned
|
|
16.
|
s3guard bucket-info command to add a verify-property <key>=<value> <bucket> |
|
Open |
Unassigned
|
|
17.
|
S3A to implement rename(final Path src, final Path dst, final Rename... options) |
|
Open |
Unassigned
|
|
18.
|
Collect AwsSdkMetrics in S3A FileSystem IOStatistics |
|
Resolved |
Steve Loughran
|
|
19.
|
Add some tests about S3 timestamp tracking |
|
Resolved |
Unassigned
|
|
20.
|
S3a operations keep retrying if the password is wrong |
|
Open |
Thomas Poepping
|
|
21.
|
Add some S3A-specific create file options |
|
Open |
Unassigned
|
|
22.
|
s3a to set fake directory marker contentType to application/x-directory |
|
Open |
Steve Loughran
|
|
23.
|
S3A Filesystem does not check return from AmazonS3Client deleteObjects |
|
Open |
Unassigned
|
|
24.
|
Optimize uses of FS operations in the ASF analysis frameworks and libraries |
|
Resolved |
Steve Loughran
|
|
25.
|
shell rm command to not rename to ~/.Trash in object stores |
|
Open |
Unassigned
|
|
26.
|
Add S3A support for Async Scatter/Gather IO |
|
Open |
Gabor Bota
|
|
27.
|
Impersonate hosts in s3a for better data locality handling |
|
Open |
Thomas Demoor
|
|
28.
|
log DNS addresses on s3a init |
|
Resolved |
Mukund Thakur
|
|
29.
|
export s3a BlockingThreadPoolExecutorService pool info (size, load) as gauges |
|
Resolved |
Unassigned
|
|
30.
|
increase the default number of threads and http connections in S3A |
|
Open |
Unassigned
|
|
31.
|
S3A authenticators to log origin of .secret.key options |
|
Resolved |
Unassigned
|
|
32.
|
Support AWS S3 reduced redundancy storage class |
|
Open |
Unassigned
|
|
33.
|
s3a doesn't consider blobs with trailing / and content-length >0 as directories |
|
Open |
Unassigned
|
|
34.
|
make s3a read fault injection configurable including "off" |
|
Open |
Unassigned
|
|
35.
|
Retrive modtime of PUT file from store, via response or HEAD |
|
Resolved |
Unassigned
|
|
36.
|
s3a create() doesn't check for an ancestor path being a file |
|
Resolved |
Sean Mackrory
|
|
37.
|
strip s3.amazonaws.com off hostnames before making s3a calls |
|
Open |
Unassigned
|
|
38.
|
add a special 0 byte input stream for empty blobs |
|
Open |
Unassigned
|
|
39.
|
improve setting of max connections in AWS client |
|
Open |
Unassigned
|
|
40.
|
multipart/huge file upload tests to look at checksums returned |
|
Open |
Unassigned
|
|
41.
|
Some S3A tests leak filesystem instances |
|
Open |
Unassigned
|
|
42.
|
s3guard to provide better diags on ddb init failures |
|
Open |
Unassigned
|
|
43.
|
Improve isolation of FS instances in S3A committer tests |
|
Open |
Unassigned
|
|
44.
|
s3a rm on the CLI generates deprecation warning on io.bytes.per.checksum |
|
Open |
Unassigned
|
|
45.
|
S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use |
|
Open |
Unassigned
|
|
46.
|
review S3A translateException translation matches IBM CORS spec |
|
Open |
Unassigned
|
|
47.
|
Improve S3A rename resilience |
|
Resolved |
Steve Loughran
|
|
48.
|
add tests/docs for HAR files on s3a |
|
Resolved |
Unassigned
|
|
49.
|
S3A to use a thread pool for async path operations |
|
Open |
Unassigned
|
|
50.
|
builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate upload |
|
Open |
Unassigned
|
|
51.
|
FileSystem/s3a processDeleteOnExit to skip the exists() check |
|
Open |
Unassigned
|
|
52.
|
Add custom InstanceProfileCredentialsProvider with more resilience to throttling |
|
Open |
Unassigned
|
|
53.
|
ITestS3AMiniYarnCluster fails on sequential runs with Kerberos error |
|
Open |
Unassigned
|
|
54.
|
S3a DelegationToken bindings to to support a "correlation ID" for the UA header |
|
Open |
Unassigned
|
|
55.
|
add experimental optimization of s3a directory marker handling |
|
Resolved |
Unassigned
|
|
56.
|
Report on S3A cached 404 recovery better |
|
Resolved |
Unassigned
|
|
57.
|
Stabilize openFile() and adopt internally |
|
In Progress |
Steve Loughran
|
|
58.
|
S3AInputStream.skip() to use lazy seek |
|
Open |
Unassigned
|
|
59.
|
s3a to instrument duration of HTTP calls |
|
Resolved |
Steve Loughran
|
|
60.
|
Document `dynamodb:TagResource` an explicit client-side permission for S3Guard |
|
Open |
Gabor Bota
|
|
61.
|
catch and downgrade all exceptions trying to load openssl native libs through wildfly |
|
Resolved |
Steve Loughran
|
|
62.
|
ITestS3ARemoteFileChanged doesn't overwrite test data creation |
|
Open |
Unassigned
|
|
63.
|
Optimize getFileStatus in S3A |
|
Resolved |
Steven K. Wong
|
|
64.
|
s3a mkdir path/ can add 404 to S3 load balancers |
|
Resolved |
Unassigned
|
|
65.
|
AWS Data read stack trace in S3a putObjectDirect |
|
Open |
Unassigned
|
|
66.
|
AmazonClient 30x exceptions to include redirect URL in message |
|
Resolved |
Unassigned
|
|
67.
|
S3A FS to add "s3a:no-existence-checks" to the builder file creation option set |
|
Open |
Unassigned
|
|
68.
|
s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet |
|
Open |
Unassigned
|
|
69.
|
Use lighter-weight alternatives to innerGetFileStatus where possible |
|
Open |
Unassigned
|
|
70.
|
S3A Input Stream bytes read counter isn't getting through to StorageStatistics/instrumentation properly |
|
Resolved |
Unassigned
|
|
71.
|
job commit failure in S3A MR magic committer test |
|
Resolved |
Steve Loughran
|
|
72.
|
intermittent failure of ITestS3GuardListConsistency.testInconsistentS3ClientDeletes in parallel runs |
|
Resolved |
Unassigned
|
|
73.
|
Add S3AWriteOpContext for write ops; pass in statistics and other settings |
|
Open |
Unassigned
|
|
74.
|
S3A: Set thread names with more specific information about the call. |
|
Open |
Unassigned
|
|
75.
|
S3A Client to add explicit support for versioned stores |
|
Resolved |
Steve Loughran
|
|
76.
|
Add a way for an FS instance to say "really, no trash interval at all" |
|
Open |
Unassigned
|
|
77.
|
s3guard uploads command to list date and initiator of outstanding uploads |
|
Open |
Unassigned
|
|
78.
|
test and document use of fs.s3a.signing-algorithm |
|
Open |
Unassigned
|
|
79.
|
S3A statistic collection underrecords bytes written in helper threads |
|
Resolved |
Steve Loughran
|
|
80.
|
S3AInputStream.seek should throw EOFException if seeking past the end of file |
|
Open |
Unassigned
|
|
81.
|
S3a auth exception to link to a wiki page on the problem |
|
Open |
Unassigned
|
|
82.
|
s3a to improve diags on s3a bad request message |
|
Open |
Unassigned
|
|
83.
|
AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade |
|
Open |
Steve Loughran
|
|
84.
|
s3a: auto-detect region for bucket and use right endpoint |
|
Resolved |
Aaron Fabbri
|
|
85.
|
ITestS3AContractRootDir failure on non-S3Guarded bucket |
|
Open |
Unassigned
|
|
86.
|
s3guard bucket-info command to include default bucket encryption info |
|
Open |
Unassigned
|
|
87.
|
S3 SSEC tests to downgrade when running against a mandatory encryption object store |
|
Open |
Unassigned
|
|
88.
|
S3AInputStream read(bytes[]) to not retry on read failure: pass action up |
|
Open |
Unassigned
|
|
89.
|
ITestS3A select tests fail if user kinited in |
|
Open |
Unassigned
|
|
90.
|
S3A add histogram metrics types for latency, etc. |
|
Open |
Sean Mackrory
|
|
91.
|
cherry pick s3 ehancements from PrestoS3FileSystem |
|
Open |
Unassigned
|
|
92.
|
S3aDelegationTokens to add accessor for tests to get at the token binding |
|
Open |
Unassigned
|
|
93.
|
s3guard LimitExceededException -too many tables |
|
Resolved |
Unassigned
|
|
94.
|
Support multipart download in S3AFileSystem |
|
Open |
Unassigned
|
|
95.
|
NPE in S3AInputStream.read() in ITestS3AInconsistency.testOpenFailOnRead |
|
Open |
Unassigned
|
|
96.
|
S3 Select Exceptions are not being converted to IOEs |
|
Open |
Unassigned
|
|
97.
|
s3a directory housekeeping operations to be done in async thread |
|
Resolved |
Unassigned
|
|
98.
|
remove misleading fs.s3a.delegation.tokens.enabled prompt |
|
Open |
Unassigned
|
|
99.
|
ITestS3ARemoteFileChanged tests fail if you set the bucket to versionid tracking |
|
Resolved |
Unassigned
|
|
100.
|
Review S3A documentation to make sure it is consistent with the current codebase |
|
Open |
Unassigned
|
|
101.
|
Clarify committers.md around v2 failure handling |
|
Open |
Unassigned
|
|
102.
|
test YARN log collection works to s3a |
|
Open |
Unassigned
|
|
103.
|
Encrypt S3A data client-side with AWS SDK (S3-CSE) |
|
Patch Available |
Igor Mazur
|
|
104.
|
Add some Java-8 friendly way to work with RemoteIterable, especially listings |
|
Resolved |
Unassigned
|
|
105.
|
Handle S3A "glacier" data |
|
Open |
Unassigned
|
|
106.
|
Add common getFileBlockLocations() emulation for object stores, including S3A |
|
Patch Available |
Steve Loughran
|
|
107.
|
S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes |
|
Resolved |
Kazuyuki Tanimura
|
|
108.
|
s3a mkdirs() to not check dest for a dir marker |
|
Resolved |
Unassigned
|
|
109.
|
clean up ITestS3AFileSystemContract |
|
Patch Available |
Unassigned
|
|
110.
|
S3A FS deleteOnExit to skip the exists check |
|
Resolved |
Unassigned
|
|
111.
|
S3A to optionally retain directory markers |
|
Resolved |
Steve Loughran
|
|
112.
|
Add some Abortable.abort() interface for streams etc which can be terminated |
|
Resolved |
Jungtaek Lim
|
|
113.
|
declare that fs.s3a.ext. is a prefix for arbitrary extensions |
|
Open |
Unassigned
|
|
114.
|
IAM role created by S3A DT doesn't include DynamoDB scan |
|
Resolved |
Unassigned
|
|
115.
|
S3A FullCredentialsTokenBinding fails if local credentials are unset |
|
Resolved |
Steve Loughran
|
|
116.
|
S3AFilesystem trash handling should respect the current UGI |
|
Open |
Unassigned
|
|
117.
|
S3AInputStream.remainingInFile should use nextReadPos |
|
Reopened |
lqjacklee
|
|
118.
|
s3guard can't init table if caller doesn't have tag permissions |
|
Resolved |
Unassigned
|
|
119.
|
Possible inconsistent state of AbstractDelegationTokenSecretManager |
|
Patch Available |
Hankó Gergely
|
|
120.
|
ITestS3GuardOutOfBandOperations.testListingDelete failing on versioned bucket |
|
Resolved |
Steve Loughran
|
|
121.
|
Add option for a prefix to put in front of every s3guard table |
|
Resolved |
Unassigned
|
|
122.
|
Add more s3guard metrics |
|
Resolved |
Unassigned
|
|
123.
|
Improve DynamoDB schema update story |
|
Resolved |
Sean Mackrory
|
|
124.
|
S3Guard: Optimize performance of handling OOB operations in non-authoritative mode |
|
Resolved |
Unassigned
|
|
125.
|
getFileChecksum() needs to adopt S3Guard |
|
Resolved |
lqjacklee
|
|
126.
|
reduce/tune read failure fault injection on inconsistent client |
|
Resolved |
Unassigned
|
|
127.
|
increase performance of s3guard import command |
|
Open |
Unassigned
|
|
128.
|
intermittent failure of ITestCommitOperations: too many s3guard writes |
|
Resolved |
Unassigned
|
|
129.
|
S3a getFileStatus to update DDB if an S3 query returns etag/versionID |
|
Resolved |
Unassigned
|
|
130.
|
Possible for modified configuration to leak into metadatastore in S3GuardTool |
|
Resolved |
Unassigned
|
|
131.
|
S3Guard instrumentation to include cost of DynamoDB ops as metric |
|
Resolved |
Unassigned
|
|
132.
|
Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations |
|
Resolved |
Unassigned
|
|
133.
|
Intermittent failure of ITestS3GuardToolDynamoDB#testDynamoDBInitDestroyCycle |
|
Resolved |
Unassigned
|
|
134.
|
S3Guard init command uses global settings, not those of target bucket |
|
Resolved |
Steve Loughran
|
|
135.
|
Improve throttling on S3Guard DDB batch retries |
|
Resolved |
Unassigned
|
|
136.
|
S3guard: add inconsistency detection metrics |
|
Resolved |
Unassigned
|
|
137.
|
S3Guard prune to only remove auth dir marker if files (not tombstones) are removed |
|
Resolved |
Unassigned
|
|
138.
|
ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store |
|
Resolved |
Unassigned
|
|
139.
|
tag S3GuardTool entry points as limitedPrivate("management-tools")/evolving |
|
Resolved |
Steve Loughran
|
|
140.
|
Fix ITestS3GuardToolLocal#testInitNegativeRead test failure |
|
Resolved |
Steve Loughran
|
|
141.
|
Ensure controls in-place to prevent clients with significant clock skews pruning aggressively |
|
Resolved |
Unassigned
|
|
142.
|
Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily restrictive |
|
Resolved |
Unassigned
|
|
143.
|
improvements to S3GuardTool destroy command |
|
Resolved |
Unassigned
|
|
144.
|
Clock skew can cause S3Guard to think object metadata is out of date |
|
Resolved |
Unassigned
|
|
145.
|
S3guard metadata stores to support millions of entries |
|
Resolved |
Unassigned
|
|
146.
|
ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle fails if test bucket isn't on demand |
|
Resolved |
Steve Loughran
|
|
147.
|
S3Guard to self update on directory listings of S3 |
|
Resolved |
Unassigned
|
|
148.
|
S3guard mistakes root URI without / as non-absolute path |
|
Resolved |
Unassigned
|
|
149.
|
HADOOP-16953. tune s3guard disabled warnings |
|
Resolved |
Steve Loughran
|
|
150.
|
mkdir on s3a should not be sensitive to trailing '/' |
|
Resolved |
Steve Loughran
|
|
151.
|
s3a to not need wildfly on the classpath |
|
Resolved |
Steve Loughran
|
|
152.
|
ITestS3AConfiguration proxy tests fail when bucket probes == 0 |
|
Resolved |
Gabor Bota
|
|
153.
|
S3A to support additional token issuers |
|
Resolved |
Steve Loughran
|
|
154.
|
S3A staging committer committing duplicate files |
|
Resolved |
Steve Loughran
|
|
155.
|
ITestS3GuardOutOfBandOperations testListingDelete[auth=false] fails on unversioned bucket |
|
Resolved |
Unassigned
|
|
156.
|
S3AFileSystem silently deletes "fake" directories when writing a file. |
|
Resolved |
Unassigned
|
|
157.
|
ITestS3AEncryptionWithDefaultS3Settings fails if default bucket encryption != KMS |
|
Resolved |
Mukund Thakur
|
|
158.
|
Handle transient stream read failures in FileSystem contract tests |
|
Resolved |
Unassigned
|
|
159.
|
add way for s3a to recognise buckets with "." in name and switch to path access |
|
Resolved |
Unassigned
|
|
160.
|
Decrease size of s3a dependencies |
|
Resolved |
Unassigned
|
|
161.
|
Backport HADOOP-13230 list/getFileStatus changes for preserved directory markers |
|
Resolved |
Steve Loughran
|
|
162.
|
Renaming a file under a sibling empty directory doesn't delete dest dir's marker |
|
Resolved |
Steve Loughran
|
|
163.
|
improve s3guard markers command line tool |
|
Resolved |
Steve Loughran
|
|
164.
|
Backport HADOOP-13230 listing changes for preserved directory markers to 3.1.x |
|
Resolved |
Steve Loughran
|
|
165.
|
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. |
|
Resolved |
Steve Loughran
|
|
166.
|
s3a rename() now requires s3:deleteObjectVersion permission |
|
Resolved |
Steve Loughran
|
|
167.
|
S3A to always probe S3 in S3A getFileStatus on non-auth paths |
|
Resolved |
Steve Loughran
|
|
168.
|
ITestCustomSigner fails with gcs s3 compatible endpoint. |
|
Resolved |
Mukund Thakur
|
|
169.
|
S3AInputStream to be resilient to faiures in abort(); translate AWS Exceptions |
|
Resolved |
Yongjun Zhang
|
|
170.
|
FileSystem.get to support slow-to-instantiate FS clients |
|
Resolved |
Steve Loughran
|
|
171.
|
S3A committer to support concurrent jobs with same app attempt ID & dest dir |
|
Resolved |
Steve Loughran
|
|
172.
|
S3A marker tool mixes up -min and -max |
|
Resolved |
Steve Loughran
|
|
173.
|
hadoop-cloud-storage transient dependencies need review |
|
Open |
Unassigned
|
|
174.
|
ITestS3AContractRename failing against stricter tests |
|
Resolved |
Attila Doroszlai
|
|
175.
|
AbstractS3ATokenIdentifier to set issue date == now |
|
Resolved |
Jungtaek Lim
|
|
176.
|
Upgrade aws-java-sdk to 1.11.901 |
|
Resolved |
Steve Loughran
|
|
177.
|
ITestS3ADeleteCost.testDirMarkersFileCreation failure |
|
Resolved |
Steve Loughran
|
|
178.
|
AbstractS3ATokenIdentifier to issue date in UTC |
|
Resolved |
Jungtaek Lim
|
|
179.
|
enable s3a magic committer by default |
|
Resolved |
Steve Loughran
|
|
180.
|
Magic committer files don't have the count of bytes written collected by spark |
|
Resolved |
Steve Loughran
|
|
181.
|
Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc |
|
Resolved |
Yongjun Zhang
|
|
182.
|
S3A docs to state s3 is consistent, deprecate S3Guard |
|
Resolved |
Steve Loughran
|
|
183.
|
magic committer to be enabled for all S3 buckets |
|
Resolved |
Steve Loughran
|
|
184.
|
typo in MagicCommitTracker |
|
Resolved |
Pierrick HYMBERT
|
|
HADOOP-15387 Produce a shaded hadoop-cloud-storage JAR for applications to use
HADOOP-15788 Improve Distcp for long-haul/cloud deployments
HADOOP-16111 FileSystem.closeAllForUGI closes() filesystem in Sync block
