[SPARK-20110] Windowed aggregation do not work when the timestamp is a nested field

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.1.0
• Component/s: Input/Output
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Alexis Seigneurin
• Created: 27/Mar/17 20:33
• Updated: 14/Nov/19 12:09
• Resolved: 21/May/19 04:11

Description

I am loading data into a DataFrame with nested fields. I want to perform a windowed aggregation on the timestamp from a nested fields:

  .groupBy(window($"auth.sysEntryTimestamp", "2 minutes"))

I get the following error:

org.apache.spark.sql.AnalysisException: Multiple time window expressions would result in a cartesian product of rows, therefore they are not currently not supported.

This works fine if I first extract the timestamp to a separate column:

  .withColumn("sysEntryTimestamp", $"auth.sysEntryTimestamp")
  .groupBy(
    window($"sysEntryTimestamp", "2 minutes")
  )

Please see the whole sample:

• batch: https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4683710270868386/4278399007363210/3769253384867782/latest.html
• Structured Streaming: https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4683710270868386/4278399007363192/3769253384867782/latest.html