Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-237

Refactor ParentTaskDataFetcher to emit streaming data and watermark

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.1

    Description

      Currently, ParentTaskDataFetcher is designed for batch jobs. 

      It retrieves multiple customized Iterators from {{InputReader, each of which has blocking hasNext()}} operation. This will block fetching data from multiple streams if a data stream does not emit data. 

      We should refactor ParentTaskDataFetcher, InputReader and related classes to retrieve streaming data and watermark without blocking. 

      Attachments

        Issue Links

          is a child of

          Umbrella - An overarching type made of sub-tasks NEMO-157 Support Nemo Streaming

          • Major - Major loss of function.
          • Resolved

          Umbrella - An overarching type made of sub-tasks NEMO-229 Support watermark

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #138

          Activity

            People

              Unassigned Unassigned
              taegeonum Tae-Geon Um
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: