[SPARK-29758] json_tuple truncates fields

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.4.4
• Fix Version/s: 2.4.5
• Component/s: SQL

Environment

EMR 5.15.0 (Spark 2.3.0) And MacBook Pro (Mojave 10.14.3, Spark 2.4.4)
Jdk 8, Scala 2.11.12

• Assignee: Max Gekk
• Reporter: Stanislav
• Created: 05/Nov/19 11:26
• Updated: 21/Nov/19 00:59
• Resolved: 20/Nov/19 07:33

Description

`json_tuple` has inconsistent behaviour with `from_json` - but only if json string is longer than 2700 characters or so.

This can be reproduced in spark-shell and on cluster, but not in scalatest, for some reason.

import org.apache.spark.sql.functions.{from_json, json_tuple}
import org.apache.spark.sql.types._
val counterstring = "*3*5*7*9*12*15*18*21*24*27*30*33*36*39*42*45*48*51*54*57*60*63*66*69*72*75*78*81*84*87*90*93*96*99*103*107*111*115*119*123*127*131*135*139*143*147*151*155*159*163*167*171*175*179*183*187*191*195*199*203*207*211*215*219*223*227*231*235*239*243*247*251*255*259*263*267*271*275*279*283*287*291*295*299*303*307*311*315*319*323*327*331*335*339*343*347*351*355*359*363*367*371*375*379*383*387*391*395*399*403*407*411*415*419*423*427*431*435*439*443*447*451*455*459*463*467*471*475*479*483*487*491*495*499*503*507*511*515*519*523*527*531*535*539*543*547*551*555*559*563*567*571*575*579*583*587*591*595*599*603*607*611*615*619*623*627*631*635*639*643*647*651*655*659*663*667*671*675*679*683*687*691*695*699*703*707*711*715*719*723*727*731*735*739*743*747*751*755*759*763*767*771*775*779*783*787*791*795*799*803*807*811*815*819*823*827*831*835*839*843*847*851*855*859*863*867*871*875*879*883*887*891*895*899*903*907*911*915*919*923*927*931*935*939*943*947*951*955*959*963*967*971*975*979*983*987*991*995*1000*1005*1010*1015*1020*1025*1030*1035*1040*1045*1050*1055*1060*1065*1070*1075*1080*1085*1090*1095*1100*1105*1110*1115*1120*1125*1130*1135*1140*1145*1150*1155*1160*1165*1170*1175*1180*1185*1190*1195*1200*1205*1210*1215*1220*1225*1230*1235*1240*1245*1250*1255*1260*1265*1270*1275*1280*1285*1290*1295*1300*1305*1310*1315*1320*1325*1330*1335*1340*1345*1350*1355*1360*1365*1370*1375*1380*1385*1390*1395*1400*1405*1410*1415*1420*1425*1430*1435*1440*1445*1450*1455*1460*1465*1470*1475*1480*1485*1490*1495*1500*1505*1510*1515*1520*1525*1530*1535*1540*1545*1550*1555*1560*1565*1570*1575*1580*1585*1590*1595*1600*1605*1610*1615*1620*1625*1630*1635*1640*1645*1650*1655*1660*1665*1670*1675*1680*1685*1690*1695*1700*1705*1710*1715*1720*1725*1730*1735*1740*1745*1750*1755*1760*1765*1770*1775*1780*1785*1790*1795*1800*1805*1810*1815*1820*1825*1830*1835*1840*1845*1850*1855*1860*1865*1870*1875*1880*1885*1890*1895*1900*1905*1910*1915*1920*1925*1930*1935*1940*1945*1950*1955*1960*1965*1970*1975*1980*1985*1990*1995*2000*2005*2010*2015*2020*2025*2030*2035*2040*2045*2050*2055*2060*2065*2070*2075*2080*2085*2090*2095*2100*2105*2110*2115*2120*2125*2130*2135*2140*2145*2150*2155*2160*2165*2170*2175*2180*2185*2190*2195*2200*2205*2210*2215*2220*2225*2230*2235*2240*2245*2250*2255*2260*2265*2270*2275*2280*2285*2290*2295*2300*2305*2310*2315*2320*2325*2330*2335*2340*2345*2350*2355*2360*2365*2370*2375*2380*2385*2390*2395*2400*2405*2410*2415*2420*2425*2430*2435*2440*2445*2450*2455*2460*2465*2470*2475*2480*2485*2490*2495*2500*2505*2510*2515*2520*2525*2530*2535*2540*2545*2550*2555*2560*2565*2570*2575*2580*2585*2590*2595*2600*2605*2610*2615*2620*2625*2630*2635*2640*2645*2650*2655*2660*2665*2670*2675*2680*2685*2690*2695*2700*2705*2710*2715*2720*2725*2730*2735*2740*2745*2750*2755*2760*2765*2770*2775*2780*2785*2790*2795*2800*"
val json_tuple_result = Seq(s"""{"test":"$counterstring"}""").toDF("json")
  .withColumn("result", json_tuple('json, "test"))
  .select('result)
  .as[String].head.length
val from_json_result = Seq(s"""{"test":"$counterstring"}""").toDF("json")
  .withColumn("parsed", from_json('json, StructType(Seq(StructField("test", StringType)))))
  .withColumn("result", $"parsed.test")
  .select('result)
  .as[String].head.length

scala> json_tuple_result
res62: Int = 2791

scala> from_json_result
res63: Int = 2800

Result is influenced by the total length of the json string at the moment of parsing:

val json_tuple_result_with_prefix = Seq(s"""{"prefix": "dummy", "test":"$counterstring"}""").toDF("json")
  .withColumn("result", json_tuple('json, "test"))
  .select('result)
  .as[String].head.length

scala> json_tuple_result_with_prefix
res64: Int = 2772

Issue Links

• is related to: SPARK-29978 Add a test to check `json_tuple` does not truncate results
• links to: GitHub Pull Request #26563