The FsPermissionCache is described as:
I confirmed, and FsPermissionCache#precacheChildrenOf is actually called for data stored on S3. The issue is that FsPermissionCache#getPermissions is called inside HdfsTable#getAvailableAccessLevel, which is skipped for S3. So all the cached metadata is not used. The problem is that precacheChildrenOf calls getFileStatus for all files, which results in a bunch of unnecessary metadata operations to S3 + a bunch of cached metadata that is never used.
precacheChildrenOf is actually only invoked in the specific scenario described below:
Regardless, skipping the call to precacheChildrenOf for blob stores should (1) improve table loading time for S3 backed tables, and (2) decrease catalogd memory requirements when loading a bunch of tables stored on S3.