Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-26388

Release Testing: Repeatable Cleanup (FLINK-25433)

    XMLWordPrintableJSON

Details

    Description

      Repeatable cleanup got introduced with FLIP-194 but should be considered as an independent feature of the JobResultStore (JRS) from a user's point of view.

      Repeatable cleanup can be triggered by running into an error while cleaning up. This can be achieved by disabling access to S3 after the job finished, e.g.:

      • Setting a reasonable enough checkpointing time (checkpointing should be enabled to allow cleanup of s3)
      • Disable s3 (removing permissions or shutting down the s3 server)
      • Stop job with savepoint

      Stopping the job should work but the logs should show failure with repeating retries. Enabling S3 again should fix the issue.

      Keep in mind that if testing this in with HA, you should use a different bucket for the file-based JRS artifacts only change permissions for the bucket that holds JRS-unrelated artifacts. Flink would fail fatally if the JRS is not able to access it's backend storage.

      Documentation and configuration is still in the process of being updated in FLINK-26296 and FLINK-26331

      Attachments

        Issue Links

          is caused by

          Sub-task - The sub-task of the issue FLINK-25433 Integrate retry strategy for cleanup stage

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue FLINK-26296 Add missing documentation

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Sub-task - The sub-task of the issue FLINK-26331 Make max retries configurable

          • Major - Major loss of function.
          • Resolved
          Testing discovered

          Bug - A problem which impairs or prevents the functions of the product. FLINK-26488 FileSystem.listFiles is not implemented consistently

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. FLINK-26606 CompletedCheckpoints that failed to be discarded are not stored in the CompletedCheckpointStore

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. FLINK-26450 FileStateHandle.discardState does not process return value

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. FLINK-26484 FileSystem.delete is not implemented consistently

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. FLINK-26494 Missing logs during retry

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Web Link GitHub Pull Request #19058

          Activity

            People

              dwysakowicz Dawid Wysakowicz
              mapohl Matthias Pohl
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: