Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18087

Uploading user artifact for Yarn job cluster could not work

    XMLWordPrintableJSON

    Details

      Description

      In FLINK-17632, we add the support remote user jar. However, uploading user artifact for Yarn job cluster is broken exceptionally. In the following code, we should only upload local files. Now it has the contrary behavior.

      // only upload local files
      if (Utils.isRemotePath(entry.getValue().filePath)) {
         Path localPath = new Path(entry.getValue().filePath);
         Tuple2<Path, Long> remoteFileInfo =
               fileUploader.uploadLocalFileToRemote(localPath, entry.getKey());
         jobGraph.setUserArtifactRemotePath(entry.getKey(), remoteFileInfo.f0.toString());
      }
      

       

      Another problem is the related tests testPerJobModeWithDistributedCache does not fail because we do not fetch the artifact from remote filesystem(i.e. HDFS). We directly get it from local file. It also needs to be enhanced.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fly_in_gis Yang Wang
                Reporter:
                fly_in_gis Yang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: