[SPARK-36337] decimal('Nan') is unsupported in net.razorvine.pickle

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Assignee: Yikun Jiang
• Reporter: Yikun Jiang
• Created: 29/Jul/21 04:02
• Updated: 12/Dec/22 18:11
• Resolved: 15/Oct/21 02:28

Description

Decimal('NaN') is not supported by net.razorvine.pickle now.

In Python

>>> pickled = cloudpickle.dumps(decimal.Decimal('NaN'))
b'\x80\x05\x95!\x00\x00\x00\x00\x00\x00\x00\x8c\x07decimal\x94\x8c\x07Decimal\x94\x93\x94\x8c\x03NaN\x94\x85\x94R\x94.'
>>> pickle.loads(pickled)
Decimal('NaN')

In Scala

scala> import net.razorvine.pickle.\{Pickler, Unpickler, PickleUtils}
scala> val unpickle = new Unpickler
scala> unpickle.loads(PickleUtils.str2bytes("\u0080\u0005\u0095!\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u008c\u0007decimal\u0094\u008c\u0007Decimal\u0094\u0093\u0094\u008c\u0003NaN\u0094\u0085\u0094R\u0094."))
net.razorvine.pickle.PickleException: problem construction object: java.lang.reflect.InvocationTargetException
 at net.razorvine.pickle.objects.AnyClassConstructor.construct(AnyClassConstructor.java:29)
 at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:773)
 at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:213)
 at net.razorvine.pickle.Unpickler.load(Unpickler.java:123)
 at net.razorvine.pickle.Unpickler.loads(Unpickler.java:136)
 ... 48 elided

I submit an issue in pickle upstream https://github.com/irmen/pickle/issues/7 .

we should bump pickle latest version after it fixed.

Issue Links

• links to: [Github] Pull Request #34285 (Yikun)
• links to: [Github] Pull Request #34314 (Yikun)