[FLINK-8971] Create general purpose testing job

• Type: New Feature
• Status: Closed
• Priority: Critical
• Resolution: Done
• Affects Version/s: 1.5.0
• Component/s: Tests
• Assignee: Unassigned
• Reporter: Till Rohrmann
• Created: 16/Mar/18 10:34
• Updated: 23/Apr/21 16:30
• Resolved: 23/Apr/21 16:30

Description

In order to write better end-to-end tests we need a general purpose testing job which comprises as many Flink aspects as possible. These include different types for records and state, user defined components, state types and operators.

The job should allow to activate a certain misbehavior, such as slowing certain paths down or throwing exceptions to simulate failures.

The job should come with a data generator which generates input data such that the job can verify it's own behavior. This includes the state as well as the input/output records.

We already have the heavily misbehaved job which simulates some misbehavior. There is also the state machine job which can verify itself for invalid state changes which indicate data loss. We should incorporate their characteristics into the new general purpose job.

Additionally, the general purpose job should contain the following aspects:

• Job containing a sliding window aggregation
• At least one generic Kryo type
• At least one generic Avro type
• At least one Avro specific record type
• At least one input type for which we register a Kryo serializer
• At least one input type for which we provide a user defined serializer
• At least one state type for which we provide a user defined serializer
• At least one state type which uses the AvroSerializer
• Include an operator with ValueState
• Value state changes should be verified (e.g. predictable series of values)
• Include an operator with operator state
• Include an operator with broadcast state
• Broadcast state changes should be verified (e.g. predictable series of values)
• Include union state
• User defined watermark assigner

The job should be made available in the flink-end-to-end-tests module.

This issue is intended to serve as an umbrella issue for developing and extending this job.

Issue Links

• is related to: FLINK-20103 Improve test coverage with chaos testing & side-by-side tests
• is related to: FLINK-22232 Improve test coverage for network stack
• relates to: FLINK-8970 Add more automated end-to-end tests
• links to: GitHub Pull Request #5941

Sub-Tasks

• Implement source and operator that validate exactly-once: Closed
• Ensure the job has an operator with operator state.: Closed
• Add a test operator with keyed state that uses Kryo serializer (registered/unregistered/custom): Resolved
• Add a test operator with keyed state that uses Avro serializer (from schema/by reflection): Resolved
• Add a test operator with keyed state that uses custom, stateful serializer: Closed
• Include an operator with broadcast and union state: Closed
• Add sliding window aggregation to the job: Closed
• Add user-defined watermark assigner and timestamp extractor to the test job: Closed
• Add operators with input type that goes through Kryo serialization (registered/generic/custom): Closed
• Add operators with input type that goes through Avro serialization (with schema/generic): Closed
• Add operators with input type that goes through custom, stateful serialization: Closed
• Add exception throwing map function that simulates failures to the general purpose DataStream job: Closed
• Assign uids to all operators: Closed