[IMPALA-1850] allow fs.defaultFS to be set to a non-HDFS filesystem

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.2
• Fix Version/s: Impala 2.6.0
• Component/s: Frontend
• Labels: ramp-up s3 usability
• Target Version: Impala 2.6.0
• Assignee: Anuj Phadke
• Reporter: Daniel Hecht
• Created: 06/Mar/15 01:33
• Updated: 25/Apr/16 03:34
• Resolved: 25/Apr/16 03:34

Description

Now that Impala can access other Hadoop filesystems, it'd be nice if those filesystems could also be used as the fs.defaultFS. For example, this would allow Impala to run against the local filesystem without needing to start an HDFS cluster.

The remaining things to do are:

1) Downgrade the checkConfiguration() checks from an error. e.g.:
a) checkFilesystem() looks for DFS.
b) checkConfiguration() checks the short circuit socket.

2) Fix HdfsCacheUtils related code. HdfsCacheUtils still assumes that the defaultFS has type DistributedFileSystem and that all cached entities will be against the defaultFS. Most code paths can infer the filesystem from the table/partition directory, but some paths such as waitForCacheDirective() will need some way to determine the filesystem.

Issue Links

• duplicates: IMPALA-393 Fix how we create the Hdfs File System object