[HIVE-21483] Fix HoS when scratch_dir is using remote HDFS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Dapeng Sun
• Reporter: Dapeng Sun
• Created: 20/Mar/19 11:56
• Updated: 20/Mar/19 12:04
• Resolved: 20/Mar/19 12:03

Description

HoS would fail when scratch dir is using remote HDFS:

  public static URI uploadToHDFS(URI source, HiveConf conf) throws IOException {
    Path localFile = new Path(source.getPath());
    Path remoteFile = new Path(SessionState.get().getSparkSession().getHDFSSessionDir(),
        getFileName(source));
-    FileSystem fileSystem = FileSystem.get(conf);
+    FileSystem fileSystem = remoteFile.getFileSystem(conf);
    // Overwrite if the remote file already exists. Whether the file can be added
    // on executor is up to spark, i.e. spark.files.overwrite
    fileSystem.copyFromLocalFile(false, true, localFile, remoteFile);
    Path fullPath = fileSystem.getFileStatus(remoteFile).getPath();

Issue Links

• duplicates: HIVE-16292 SparkUtilities upload to HDFS doesn't work with viewfs