Uploaded image for project: 'StreamPipes'
  1. StreamPipes
  2. STREAMPIPES-292

Introduce event windowing to the StreamPipes core/sdk

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Backend

    Description

      Apache StreamPipes

      Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.

      Background

      Currently, window logic can be individually defined per pipeline element. The whole windowing logic needs to be declared in the controller and runtime logic needs to be individually added based on the selected runtime wrapper (Java, Siddhi, Flink, etc...).

      As many data processors benefit from using window-functions (i.e PEs such as Event Counter, Count Aggregation, Rate Limiter), windowing logic is often duplicated as it needs to be implemented for every new pipeline element. In addition, the feature set of supported window operators differs (and often depends on the developer) as it is unclear which windows and parameters should/can be offered.  

      Therefore, adding support for explicit window semantics to the SDK/Core would make implementing data processors and sinks using windows much easier and less error-prone.

      Tasks

      1. Design and introduce new processor and controller classes for windowed event processors (e.g., WindowedDataProcessor) which handle the windowing logic internally and only expose the higher-level methods to users (i.e onCurrentEvent, onExpiredEvent, etc...).
      2. Implement internal logic for few window functions (i.e TimeWindow, LengthWindow, TimeBatchWindow, LengthBatchWindow, etc...)
      3. Write a few sample pipeline-elements using your new API!

      Relevant Skills

      • Basic knowledge in StreamPipes core (cloning the repo, going through the codebase/documents would do).
      • Basic knowledge of stream analytics window functions (this is not a must, but it's awesome if you know your way around analytics window functions).
      • Some Java experience.

      Learning Material

      For StreamPipes:

      For Streaming Analytics:

      For the context for the issue:

      Mentor

      • Grainier Perera (grainier [at]¬†apache.org).

      Attachments

        Activity

          People

            Unassigned Unassigned
            grainier Grainier Perera
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated: