Hive and a bit of Spark use getContentSummary() to get some summary stats of a filesystem. This is very expensive on S3A (and any other object store), especially as the base implementation does the recursive tree walk.
HADOOP-13208, we have a full enumeration of files under a path without directory costs...S3A can/should switch to this to speed up those places where the operation is called.
- API call needs FS spec and contract tests
- S3A could instrument invocation, so as to enable real-world popularity to be measured