[SPARK-34243] Update the spark libraries to expose Java module names

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.1
• Component/s: Java API

Environment

Making use of a `module-info.java` with the application based out of Spark would result in failure to compile the existing code for the lack of underlying packages being visible anymore to them.

• Assignee: Unassigned
• Reporter: Naman Nigam
• Created: 26/Jan/21 10:06
• Updated: 26/Jan/21 10:09

Description

Trying to derive the automatic module name for any of the spark related artifacts currently fails with the following error. 

{{naman.nigam jar --file=/Users/naman.nigam/.m2/repository/org/apache/spark/spark-core_2.12/3.1.0/spark-core_2.12-3.1.0.jar --describe-module}}

Unable to derive module descriptor for: /Users/naman.nigam/.m2/repository/org/apache/spark/spark-core_2.12/3.1.0/spark-core_2.12-3.1.0.jarspark.core.2.12: Invalid module name: '2' is not a Java identifier

 

I was hoping there could be an improvement in this direction such that the artifacts could be used in the modularised applications even if they are not themself modularised.

 

This relates to a previous bug reported under https://issues.apache.org/jira/browse/SPARK-23543, but it has been unfortunately closed as Incomplete and there seems no other way to track the progress that Spark is planning to make in this direction.