[STREAMPIPES-174] New Python Wrapper

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Component/s: Pipeline Elements
• Labels: gsoc2021 mentor streampipes
• Assignee: Unassigned
• Reporter: Patrick Wiener
• Created: 19/Jul/20 13:04
• Updated: 26/Nov/22 13:33

Description

Apache StreamPipes

Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.

 

Background

Current wrappers such as standalone (JVM, Siddhi) or distributed (Flink) already allow to develop new processors in the given runtime environment. The idea is to extend the list of standalone runtime wrappers to also support pure Python processors. We already got a minimal working version that however is pretty inflexible and still relies on Java as a proxy to the pipeline management in the backend service for the model declaration in the setup phase ( capabilities, requirements, static properties of a processor) as well as the actual invocation in the execution phase ( receiving specific configuration from pipeline management when pipeline is started). This issue is to track the status of the development.

 

Tasks

1. Add API endpoints as an interface for registration/invocation ( partly done)
2. Port relevant model classes over to Python (declaration + invocation descriptions)
3. Implement support for various transport protocols and transport formats
4. Implement dev friendly alternative to Java builder pattern for model declaration 
5. Implement overall runtime logic for Python wrapper

 

Relevant Skills

0. Don't be afraid! We'll guide you through your first steps with StreamPipes.

1. Excellent Python skills
2. Excellent understanding of stream processing paradigm incl.  message broker such as Kafka, MQTT, etc
3. Good Understanding of RESTful web services (HTTP, etc.)
4. Basic Java skills to understand existing wrapper logic

 

Info

• SIP-02 to collect design decisions https://cwiki.apache.org/confluence/display/STREAMPIPES/SIP-02+Python+wrapper 
• Current python runtime wrapper implementation: https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper-python
• POC example: https://github.com/apache/incubator-streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/python

 

Mentor

Patrick Wiener, PPMC Apache StreamPipes ([e-mail])

Issue Links

• Blocked: STREAMPIPES-175 Port current POC to core repository
• Blocked: STREAMPIPES-179 Cannot start multiple instances of processor

Sub-Tasks

• Port current POC to core repository: Resolved
• Create running example in streampipes-example repo: Resolved
• Cannot start multiple instances of processor: Resolved
• Add lightweight model declaration: Closed
• Add support for various transport protocols and formats: Closed
• Add CI for python package and upload to pypi: Open
• Write consul helper classes for service and config registration: Resolved
• Create REST endpoints for in python wrapper: Closed