Consider a table partitioned by 2 keys (dt, region). Say a dt partition could have 10000 associated regions. Consider that the user does:
As things stand now, StorageBasedAuthProvider will make individual DistributedFileSystem.listStatus() calls for each partition-directory, and authorize each one separately. It'd be faster to batch the calls, and examine multiple FileStatus objects at once.