Uploaded image for project: 'Apache Gearpump'
  1. Apache Gearpump
  2. GEARPUMP-249

reduce is not executed when following flatMap in Stream DSL

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.8.2
    • 0.8.3
    • streaming
    • None

    Description

      The issue can be reproduced if a redundant map is added between groupBy and sum

          app.source(data.lines.toList, 1, "source").
            // word => (word, count)
            flatMap(line => line.split("[\\s]+")).map((_, 1)).
            // (word, count1), (word, count2) => (word, count1 + count2)
            groupByKey().sum.log
      

      The problem is with AndThen#process method, where flatMap is lazily/ never executed.

        override def process(value: IN): TraversableOnce[OUT] = {
          first.process(value).flatMap(second.process)
        }
      

      Attachments

        Activity

          People

            mauzhang Manu Zhang
            mauzhang Manu Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: