Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21640

Method mode with String parameters within DataFrameWriter is error prone

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      The following method:

      def mode(saveMode: String): DataFrameWriter[T]
      

      sets the SaveMode of the DataFrameWriter depending on the string that is pass-in as parameter.

      There is a java Enum with all the save modes which are Append, Overwrite, ErrorIfExists and Ignore. In my current project I was writing some code that was using this enum to get the string value that I use to call the mode method:

        private[utils] val configModeAppend = SaveMode.Append.toString.toLowerCase
        private[utils] val configModeErrorIfExists = SaveMode.ErrorIfExists.toString.toLowerCase
        private[utils] val configModeIgnore = SaveMode.Ignore.toString.toLowerCase
        private[utils] val configModeOverwrite = SaveMode.Overwrite.toString.toLowerCase
      

      The configModeErrorIfExists val contains the value "errorifexists" and when I call the saveMode method using this string it does not match. I suggest to include "errorifexists" as a right match for the ErrorIfExists SaveMode.

      Will create a PR to address this issue ASAP.

        Attachments

          Activity

            People

            • Assignee:
              ardlema Alberto
              Reporter:
              ardlema Alberto
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: