[HIVE-16292] SparkUtilities upload to HDFS doesn't work with viewfs

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Hadoop Flags: Reviewed
• Assignee: Jimmy Xiang
• Reporter: Jimmy Xiang
• Created: 24/Mar/17 16:20
• Updated: 20/Mar/19 12:04
• Resolved: 27/Mar/17 16:33

Description

If scratchdir is set to viewfs, HoS fails with exception like

{noformt}

java.lang.IllegalArgumentException: Wrong FS: viewfs://ns-default/tmp/hive_scratch/hive/hive/_spark_session_dir/f4031fca-2885-4e7a-9b05-764d25d0e488/hive-exec-1.1.0-cdh5.7.2.jar, expected: hdfs://nameservice1
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:499)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1949)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.uploadToHDFS(SparkUtilities.java:86)

Issue Links

• is duplicated by: HIVE-21483 Fix HoS when scratch_dir is using remote HDFS