Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23729

Glob resolution breaks remote naming of files/archives

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.1, 2.4.0
    • Spark Submit

    Description

      Given one uses spark-submit with either of the --archives or the --files parameters, in case the file name actually contains glob patterns, the rename part (...#nameAs) of the filename will eventually be ignored.

      Thinking over the resolution cases, if the resolution results in multiple files, it does not make sense to send all of them under the same remote name. So this should then result in an error.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            misutoth Mihaly Toth
            misutoth Mihaly Toth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment