[HADOOP-18659] ABFS to add Auditing support and attach http referrer to requests

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.5
• Component/s: fs/azure
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 09/Mar/23 14:19
• Updated: 09/Mar/23 14:31

Description

the s3a auditing feature of HADOOP-17511 is wonderful in production, as we can

• scan for all IO done by a single user, job, operation
• get a comprehensive view of what was done to the store, in the order it saw it, which doesn't always match the order the client logged it, especially logged outcomes as that only happens when the threads get scheduled.

the abfs TracingContext effectively does most of this. All that is needed is to integrate with the hadoop-common audit interfaces/apis

• TracingContext to pick up global/thread-local audit contexts in construction and so extract information (tool class; principal; spark job id. etc)

The manifest committer already sets task id on the active thread whenever entered; this persists through all the IO done by the spark worker.

Issue Links

• relates to: HADOOP-17511 Add an Audit plugin point for S3A auditing/context
• links to: abfs log format