[SPARK-20432] Unioning two identical Streaming DataFrames fails during attribute resolution

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.1.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Burak Yavuz
• Created: 21/Apr/17 17:51
• Updated: 03/May/17 15:57
• Resolved: 03/May/17 15:57

Description

To reproduce, try unioning two identical Kafka Streams:

df = spark.readStream.format("kafka")... \
  .select(from_json(col("value").cast("string"), simpleSchema).alias("parsed_value"))

df.union(df).writeStream...

Exception is confusing:

org.apache.spark.sql.AnalysisException: resolved attribute(s) value#526 missing from value#511,topic#512,partition#513,offset#514L,timestampType#516,key#510,timestamp#515 in operator !Project [jsontostructs(...) AS parsed_value#357];

Issue Links

• relates to: SPARK-20441 Within the same streaming query, one StreamingRelation should only be transformed to one StreamingExecutionRelation
• links to: [Github] Pull Request #17735 (lw-lin)