Major Description
Currently ArrayConstructor handles an array of typecode 'l' as int when converting Python object in Python 2 into Java object, so if the value is larger than Integer.MAX_VALUE or smaller than Integer.MIN_VALUE then the overflow occurs.
import array data = [Row(l=array.array('l', [-9223372036854775808, 0, 9223372036854775807]))] df = spark.createDataFrame(data) df.show(truncate=False)
+----------+ |l | +----------+ |[0, 0, -1]| +----------+
This should be:
+----------------------------------------------+ |l | +----------------------------------------------+ |[-9223372036854775808, 0, 9223372036854775807]| +----------------------------------------------+
