Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32832

Use CaseInsensitiveMap for DataStreamReader/Writer options

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.6, 3.0.1
    • Fix Version/s: 2.4.8, 3.0.2, 3.1.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      When a user have multiple options like path, paTH, and PATH for the same key path, option/options is non-deterministic because extraOptions is HashMap. This issue aims to use CaseInsensitiveMap instead of HashMap to fix this bug fundamentally.

      spark.readStream
        .option("paTh", "1")
        .option("PATH", "2")
        .option("Path", "3")
        .option("patH", "4")
        .option("path", "5")
        .load()
      ...
      org.apache.spark.sql.AnalysisException:
      Path does not exist: file:/.../1;
      

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              dongjoon Dongjoon Hyun Assign to me
              Reporter:
              dongjoon Dongjoon Hyun

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment