[SPARK-32583] PySpark Structured Streaming Testing Support

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.4.4
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Felix Kizhakkel Jose
• Created: 10/Aug/20 23:37
• Updated: 12/Dec/22 18:10
• Resolved: 11/Aug/20 04:09

Description

Hello,

I have investigated a lot but couldn't get any help or resource on how to test my pyspark Structured Streaming pipeline job (ingesting from Kafka topics to S3) and how to build  Continuous Integration (CI)/Continuous Deployment (CD).

1. Is it possible to test (unit test, integration test) pyspark structured stream?

2. How to build  Continuous Integration (CI)/Continuous Deployment (CD)?