[SPARK-24548] JavaPairRDD to Dataset<Row> in SPARK generates ambiguous results

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Java API, SQL

Environment

Using Windows 10, on 64bit machine with 16G of ram.

• Assignee: L. C. Hsieh
• Reporter: Jackson
• Created: 13/Jun/18 11:39
• Updated: 18/Jun/18 18:02
• Resolved: 18/Jun/18 18:01

Description

I have data in below JavaPairRDD :

JavaPairRDD<String,Tuple2<String,String>> MY_RDD;

I tried using below code:

Encoder<Tuple2<String, Tuple2<String,String>>> encoder2 =
Encoders.tuple(Encoders.STRING(), Encoders.tuple(Encoders.STRING(),Encoders.STRING()));
Dataset<Row> newDataSet = spark.createDataset(JavaPairRDD.toRDD(MY_RDD),encoder2).toDF("value1","value2");

newDataSet.printSchema();

root
{{ |-- value1: string (nullable = true)}}
{{ |-- value2: struct (nullable = true)}}
{{ | |-- value: string (nullable = true)}}
{{ | |-- value: string (nullable = true)}}

But after creating a StackOverflow question ("https://stackoverflow.com/questions/50834145/javapairrdd-to-datasetrow-in-spark"), i got to know that values in tuple should have distinguish field names, where in this case its generating same name. Cause of this I cannot select specific column under value2.

Issue Links

• links to: [Github] Pull Request #21576 (viirya)