[HADOOP-18185] tune logging of prefetch problems

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 29/Mar/22 12:31
• Updated: 11/Nov/22 10:58

Description

we need to think what we want to do here.

• we would want errors to be counted and included in stream iostats
• but we've seen abfs apps flooded with stack traces when a transient network error breaks every prefetch.
• logging through LogExactlyOnce would highlight a problem, but on a long lived process, not show enough, unless the cache manager had a per instance value.

1. PrefetchTask needs to be non static and update the CachingBlockManager when there is a problem; it can LogExactlyOnce
2. stream statistics need to include cache values (hits, misses, prefech failures, how long futures were queued before execution...)