[HADOOP-17469] IOStatistics Phase II

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.1
• Component/s: fs, fs/azure, fs/s3
• Assignee: Mehakmeet Singh
• Reporter: Steve Loughran
• Created: 14/Jan/21 14:04
• Updated: 27/Jul/22 13:31

Description

Continue IOStatistics development with goals of

• Easy adoption in applications
• better instrumentation in hadoop codebase (distcp?)
• more stats in abfs and s3a connectors

A key has to be a thread level context for statistics so that app code doesn't have to explicitly ask for the stats for each worker thread. Instead

filesystem components update the context stats as well as thread stats (when?) and then apps can pick up.

• need to manage performance by minimising inefficient lookups, lock acquisition etc on what should be memory-only ops (read()), (write()),
• and for duration tracking, cut down on calls to System.currentTime() so that only 1 should be made per operation,
• need to propagate the context into worker threads

Target uses

• Impala
• Spark via SPARK-29397
• S3A committers
• Iceberg.

I have a WiP Parquet branch too, to see what can be done there. This shows up how the thread context is needed as its unworkable to build up your own stats shapshot. Even if you collect it for listX and stream reads, it doesn't include FS operations (e.g. rename()) and you need to rework all your methods to pass the stats collector around

Issue Links

• is blocked by: HADOOP-13551 Collect AwsSdkMetrics in S3A FileSystem IOStatistics
• supercedes: HADOOP-16830 Add Public IOStatistics API

Sub-Tasks

• MR FileInput/Output formats to aggregate IOStatistics: Resolved
• Add thread-level IOStatistics Context: Resolved
• s3guard tool dumpStorageStatistics to move to IOStatistics: Open
• Improve S3A upload statistics collection from ProgressEvent callbacks: Open
• Implement Incremental globStatus() -> RemoteIterator class: Open
• FSDataInput/Output Streams to automatically collect IOStatistics: Resolved
• Collect more S3A IOStatistics: Resolved
• ABFS to collect IOStatistics: Resolved
• FileSystem.close() to optionally log IOStats; save to local dir: Resolved
• S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close(): Resolved
• FilterFileSystem to implement and forward getIOStatistics(): Open
• IOStatistisSource to add new adder methods: Open
• Add ListWithIOStats<T> wrapper to return IOStats from a list.: Resolved
• JsonSerDeser to collect IOStatistics: Resolved
• IOStatisticsContext tuning: Resolved
• DistCP: Aggregate IOStatistics Counters in MapReduce Counters: Open
• ABFS: Add thread-level IOStatistics in ABFS streams: Open
• Make IOStatisticsStore and binding APIs public for use beyond our code: Open