[SPARK-35691] addFile/addJar/addDirectory should put CanonicalFile

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Kevin Su
• Reporter: Kevin Su
• Created: 09/Jun/21 06:24
• Updated: 16/Jun/21 00:40
• Resolved: 16/Jun/21 00:40

Description

I met the error below.
 

 21/06/07 00:06:57 ERROR SparkContext: Failed to add file:/home/runner/work/spark/spark/./core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar to Spark environment    java.lang.IllegalArgumentException: requirement failed: File spark-core_2.12-3.2.0-SNAPSHOT-tests.jar was already registered with a different path (old path = /home/runner/work/spark/spark/core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar, new path = /home/runner/work/spark/spark/./core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar 

But actually, /home/runner/work/spark/spark/./core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar* and * /*home/runner/work/spark/spark/core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar are the same*.

I think we should put the Canonical File in ConcurrentHashMap.

https://github.com/apache/spark/blob/9f010a8eb20502292b3bca42d17ce1dc357343b1/core/src/main/scala/org/apache/spark/rpc/netty/NettyStreamManager.scala#L68-L89 
 

Issue Links

• links to: [Github] Pull Request #32845 (pingsutw)
• links to: [Github] Pull Request #32845 (pingsutw)