Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.3.1
Description
Add a way for auditing tools to correlate S3 object calls with Hadoop FS API calls.
Initially just to log/forward to an auditing service.
Later: let us attach them as parameters in S3 requests, such as opentrace headeers or (my initial idea: http referrer header -where it will get into the log)
Challenges
- ensuring the audit span is created for every public entry point. That will have to include those used in s3guard tools, some defacto public APIs
- and not re-entered for active spans. s3A code must not call back into the FS API points
- Propagation across worker threads
Documentation Links
Attachments
Issue Links
- causes
-
HADOOP-17822 fs.s3a.acl.default not working after S3A Audit feature added
- Resolved
-
HADOOP-18091 S3A auditing leaks memory through ThreadLocal references
- Resolved
-
HADOOP-18094 Disable S3A auditing by default.
- Resolved
- contains
-
HADOOP-16911 S3A mkdirs to indicate which parent path element refers to a file
- Resolved
-
HADOOP-17470 Collect more S3A IOStatistics
- Resolved
-
HADOOP-16613 s3a to set fake directory marker contentType to application/x-directory
- Resolved
-
HADOOP-17380 ITestS3AContractSeek.teardown closes FS before superclass does its cleanup
- Resolved
- incorporates
-
HADOOP-17498 Add a MkdirOperation for chained S3 operations during mkdir
- Resolved
- is depended upon by
-
HADOOP-14661 S3A to support Requester Pays Buckets
- Resolved
- is related to
-
HADOOP-18659 ABFS to add Auditing support and attach http referrer to requests
- Open
- links to