[IMPALA-10711] Allow restricting the filesystem permissions check at startup to a particular directory

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.1.0
• Component/s: Frontend
• Epic Color: ghx-label-5
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 20/May/21 18:15
• Updated: 11/Jun/21 02:44
• Resolved: 11/Jun/21 02:44

Description

At startup, Impala validates that it has access to the filesystem (via JniFrontend::checkFileSystem()). It does this by doing a getTotalNumVisibleFiles() on the root path. The root path is defined by the fs.defaultFS property. On S3 and other object stores, that is always the root of the bucket.

Code:

https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/service/JniFrontend.java#L887

For security reasons, a user may want to restrict an Impala instance to a subdirectory of the s3 bucket (or other object store). This check at the root on startup makes that impossible.

Impala should provide a way to customize where the check takes place. The check is primarily about making sure that Impala has appropriate permissions and is configured correctly. Checking a subdirectory of the s3 bucket is just as effective for this purpose.