Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-112

Using non-IntervalWindows seems to fail

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P3
    • Resolution: Duplicate
    • None
    • Not applicable
    • runner-spark
    • None

    Description

      See here for more details: http://stackoverflow.com/questions/35993777/globalwindow-cannot-be-cast-to-intervalwindow

      The linked stack trace indicates this is using the Spark Runner:

      Exception
      java.lang.ClassCastException: com.google.cloud.dataflow.sdk.transforms.windowing.GlobalWindow cannot be cast to com.google.cloud.dataflow.sdk.transforms.windowing.IntervalWindow
          at com.google.cloud.dataflow.sdk.transforms.windowing.IntervalWindow$IntervalWindowCoder.encode(IntervalWindow.java:171)
          at com.google.cloud.dataflow.sdk.coders.IterableLikeCoder.encode(IterableLikeCoder.java:113)
          at com.google.cloud.dataflow.sdk.coders.IterableLikeCoder.encode(IterableLikeCoder.java:59)
          at com.google.cloud.dataflow.sdk.util.WindowedValue$FullWindowedValueCoder.encode(WindowedValue.java:599)
          at com.google.cloud.dataflow.sdk.util.WindowedValue$FullWindowedValueCoder.encode(WindowedValue.java:540)
          at com.cloudera.dataflow.spark.CoderHelpers.toByteArray(CoderHelpers.java:48)
          at com.cloudera.dataflow.spark.CoderHelpers$3.call(CoderHelpers.java:134)
          at com.cloudera.dataflow.spark.CoderHelpers$3.call(CoderHelpers.java:131)
          at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
          at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
          at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:149)
          at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
          at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
          at org.apache.spark.scheduler.Task.run(Task.scala:89)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      

      It seems likely that at some point the Spark runner is assuming that all windows are IntervalWindows, which may not be true. Specifically the GlobalWindow+Triggers is valid, as is any custom implementation of BoundedWindow.

      Attachments

        1. test_stocks.csv
          0.3 kB
          Amit Sela
        2. Testing123.java
          3 kB
          Amit Sela

        Issue Links

          Activity

            People

              amitsela Amit Sela
              bchambers Ben Chambers
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: