[HIVE-20515] Empty query results when using results cache and query temp dir, results cache dir in different filesystems

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Jason Dere
• Reporter: Jason Dere
• Created: 07/Sep/18 00:17
• Updated: 17/Nov/22 08:54
• Resolved: 09/Sep/18 05:07

Description

If the scratchdir for temporary query results and the results cache dir are in different filesystems, moving the query from the temp directory to results cache will fail.

Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, I see the following issues:

• FileSystem.rename() is used, which only works if the files are on the same filesystem. Need to use something like Hive.mvFile or something similar which can work between different filesystems.
• The return code from rename() was not checked which might possibly have caught the error here. This may not be applicable if a different method from FS.rename() is used in the proper fix.

With some filesystems (noticed this with WASB), if FileSystem.rename() returns false on failure rather than throwing an exception, then this results in empty results showing up for the query because the return code was not checked properly.