[SPARK-32756] Fix CaseInsensitiveMap in Scala 2.13

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: Karol Chmist
• Reporter: Karol Chmist
• Created: 31/Aug/20 16:58
• Updated: 02/Sep/20 13:27
• Resolved: 02/Sep/20 13:27

Description

 

"Spark SQL" module doesn't compile in Scala 2.13:

[info] Compiling 26 Scala sources to /home/karol/workspace/open-source/spark/sql/core/target/scala-2.13/classes...
[error] /home/karol/workspace/open-source/spark/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala:121: value += is not a member of org.apache.spark.sql.catalyst.util.CaseInsensitiveMap[String]
[error] Expression does not convert to assignment because:
[error] type mismatch;
[error] found : scala.collection.immutable.Map[String,String]
[error] required: org.apache.spark.sql.catalyst.util.CaseInsensitiveMap[String]
[error] expansion: this.extraOptions = this.extraOptions.+(key.$minus$greater(value))
[error] this.extraOptions += (key -> value)
[error] ^
[error] /home/karol/workspace/open-source/spark/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala:132: value += is not a member of org.apache.spark.sql.catalyst.util.CaseInsensitiveMap[String]
[error] Expression does not convert to assignment because:
[error] type mismatch;
[error] found : scala.collection.immutable.Map[String,String]
[error] required: org.apache.spark.sql.catalyst.util.CaseInsensitiveMap[String]
[error] expansion: this.extraOptions = this.extraOptions.+(key.$minus$greater(value))
[error] this.extraOptions += (key -> value)
[error] ^
[error] /home/karol/workspace/open-source/spark/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala:294: value += is not a member of org.apache.spark.sql.catalyst.util.CaseInsensitiveMap[String]
[error] Expression does not convert to assignment because:
[error] type mismatch;
[error] found : scala.collection.immutable.Map[String,String]
[error] required: org.apache.spark.sql.catalyst.util.CaseInsensitiveMap[String]
[error] expansion: this.extraOptions = this.extraOptions.+("path".$minus$greater(path))
[error] Error occurred in an application involving default arguments.
[error] this.extraOptions += ("path" -> path)
[error] ^
[error] /home/karol/workspace/open-source/spark/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala:317: type mismatch;
[error] found : Iterable[(String, String)]
[error] required: java.util.Map[String,String]
[error] Error occurred in an application involving default arguments.
[error] val dsOptions = new CaseInsensitiveStringMap(options.asJava)
[error] ^
[info] Iterable[(String, String)] <: java.util.Map[String,String]?
[info] false
[error] /home/karol/workspace/open-source/spark/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala:412: value += is not a member of org.apache.spark.sql.catalyst.util.CaseInsensitiveMap[String]
[error] Expression does not convert to assignment because:
[error] type mismatch;
[error] found : scala.collection.immutable.Map[String,String]
[error] required: org.apache.spark.sql.catalyst.util.CaseInsensitiveMap[String]
[error] expansion: DataFrameWriter.this.extraOptions = DataFrameWriter.this.extraOptions.+(DataSourceUtils.PARTITIONING_COLUMNS_KEY.$minus$greater(DataSourceUtils.encodePartitioningColumns(columns)))
[error] extraOptions += (DataSourceUtils.PARTITIONING_COLUMNS_KEY ->
[error] ^
[error] /home/karol/workspace/open-source/spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFiltersBase.scala:85: type mismatch;
[error] found : scala.collection.MapView[String,OrcFiltersBase.this.OrcPrimitiveField]
[error] required: Map[String,OrcFiltersBase.this.OrcPrimitiveField]
[error] CaseInsensitiveMap(dedupPrimitiveFields)
[error] ^
[info] scala.collection.MapView[String,OrcFiltersBase.this.OrcPrimitiveField] <: Map[String,OrcFiltersBase.this.OrcPrimitiveField]?
[info] false
[error] /home/karol/workspace/open-source/spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala:64: type mismatch;
[error] found : Iterable[(String, String)]
[error] required: java.util.Map[String,String]
[error] new CaseInsensitiveStringMap(withoutPath.asJava)
[error] ^
[info] Iterable[(String, String)] <: java.util.Map[String,String]?
[error] 7 errors found

 

The + function in CaseInsensitiveStringMap missing, resulting in + from standard Map being used, which returns Map instead of CaseInsensitiveStringMap.

Issue Links

• is caused by: SPARK-32364 Use CaseInsensitiveMap for DataFrameReader/Writer options
• links to: [Github] Pull Request #29584 (karolchmist)
• links to: [Github] Pull Request #29584 (karolchmist)