Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8971

Create general purpose testing job

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.5.0
    • Fix Version/s: None
    • Component/s: Tests
    • Labels:
      None

      Description

      In order to write better end-to-end tests we need a general purpose testing job which comprises as many Flink aspects as possible. These include different types for records and state, user defined components, state types and operators.

      The job should allow to activate a certain misbehavior, such as slowing certain paths down or throwing exceptions to simulate failures.

      The job should come with a data generator which generates input data such that the job can verify it's own behavior. This includes the state as well as the input/output records.

      We already have the heavily misbehaved job which simulates some misbehavior. There is also the state machine job which can verify itself for invalid state changes which indicate data loss. We should incorporate their characteristics into the new general purpose job.

      Additionally, the general purpose job should contain the following aspects:

      • Job containing a sliding window aggregation
      • At least one generic Kryo type
      • At least one generic Avro type
      • At least one Avro specific record type
      • At least one input type for which we register a Kryo serializer
      • At least one input type for which we provide a user defined serializer
      • At least one state type for which we provide a user defined serializer
      • At least one state type which uses the AvroSerializer
      • Include an operator with ValueState
      • Value state changes should be verified (e.g. predictable series of values)
      • Include an operator with operator state
      • Include an operator with broadcast state
      • Broadcast state changes should be verified (e.g. predictable series of values)
      • Include union state
      • User defined watermark assigner

      The job should be made available in the flink-end-to-end-tests module.

      This issue is intended to serve as an umbrella issue for developing and extending this job.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                trohrmann Till Rohrmann
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m