Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
3.3.0
-
None
-
None
Description
This 2.14.0 release is still a few weeks a way. There is an rc2 release but there will probably be an rc3 before a full release.
The reason I marked the Jira component as 'Java API' is that this issue will affect Java users more than Scala users.
I raised this separately to SPARK-40666 because I have a different reason for this issue. SPARK-40666 can probably already be closed as the CVE is fixed in jackson-databind 2.13.4.2.
There are performance issues in jackson-module-scala 2.13.x that may affect some Spark users. Specifically, the per issue is https://github.com/FasterXML/jackson-module-scala/issues/576
Scala3 support added in jackson-module-scala 2.13.0 means that if you use Scala 2.13, you should be able to use Scala3 compiled classes with jackson-module-scala. Scala3 compiled classes are harder to recognise using runtime reflection (and Jackson is built around runtime reflection). Scala2 compiled classes have specific annotations. With Scala3 compiled classes, we need to look for .tasty files. This lookup can be slow if you have a lot of jars (or big jars). Issue 576 fixes an issue where this .tasty lookup is done every time you try to serialize/deserialize a Java class with an ObjectMapper that has the DefaultScalaModule registered. I will also disable the .tasty file lookups for Scala 2.11/2.12 as they are not useful for those users.
For Spark usage, it may be worth turning off this .tasty file support altogether. This is another enhancement in jackson-module-scala (but not in the RC2 release).
I will follow up and update this issue when the v2.14.0 release is ready. This change will require updating all Jackson jars to v2.14.0 (as Jackson does not support using version mismatches - except at patch version level).
Attachments
Issue Links
- duplicates
-
-
- Resolved
-
- relates to
-
-
- Closed
-