Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2499

Support Custom Windows in Spark runner

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: runner-spark
    • Labels:
      None

      Description

      If we extend IntervalWindow and we try to merge these custom windows like in this PR:
      https://github.com/apache/beam/pull/3286
      Then spark runner fails with

      org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.ClassCastException: org.apache.beam.sdk.transforms.windowing.IntervalWindow cannot be cast to org.apache.beam.sdk.transforms.windowing.MergingCustomWindowsTest$CustomWindow

      It seems to be because of the cast to IntervalWindow there: https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java#L111

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                echauchot Etienne Chauchot
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: