[FLINK-16423] test_ha_per_job_cluster_datastream.sh gets stuck

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Duplicate
• Component/s: Runtime / Checkpointing, Tests
• Labels: pull-request-available test-stability
• Assignee: Robert Metzger
• Reporter: Robert Metzger
• Created: 04/Mar/20 13:34
• Updated: 29/Apr/20 13:19
• Resolved: 20/Apr/20 16:04

Description

This was seen in https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5905&view=logs&j=b1623ac9-0979-5b0d-2e5e-1377d695c991&t=e7804547-1789-5225-2bcf-269eeaa37447 ... the relevant part of the logs is here:

2020-03-04T11:27:25.4819486Z ==============================================================================
2020-03-04T11:27:25.4820470Z Running 'Running HA per-job cluster (rocks, non-incremental) end-to-end test'
2020-03-04T11:27:25.4820922Z ==============================================================================
2020-03-04T11:27:25.4840177Z TEST_DATA_DIR: /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-25482960156
2020-03-04T11:27:25.6712478Z Flink dist directory: /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
2020-03-04T11:27:25.6830402Z Flink dist directory: /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
2020-03-04T11:27:26.2988914Z Starting zookeeper daemon on host fv-az655.
2020-03-04T11:27:26.3001237Z Running on HA mode: parallelism=4, backend=rocks, asyncSnapshots=true, and incremSnapshots=false.
2020-03-04T11:27:27.4206924Z Starting standalonejob daemon on host fv-az655.
2020-03-04T11:27:27.4217066Z Start 1 more task managers
2020-03-04T11:27:30.8412541Z Starting taskexecutor daemon on host fv-az655.
2020-03-04T11:27:38.1779980Z Job (00000000000000000000000000000000) is running.
2020-03-04T11:27:38.1781375Z Running JM watchdog @ 89778
2020-03-04T11:27:38.1781858Z Running TM watchdog @ 89779
2020-03-04T11:27:38.1783272Z Waiting for text Completed checkpoint [1-9]* for job 00000000000000000000000000000000 to appear 2 of times in logs...
2020-03-04T13:21:29.9076797Z ##[error]The operation was canceled.
2020-03-04T13:21:29.9094090Z ##[section]Finishing: Run e2e tests

The last three lines indicate that the test is waiting forever for a checkpoint to appear.

Issue Links

• is broken by: FLINK-16770 Resuming Externalized Checkpoint (rocks, incremental, scale up) end-to-end test fails with no such file
• is duplicated by: FLINK-16770 Resuming Externalized Checkpoint (rocks, incremental, scale up) end-to-end test fails with no such file
• links to: GitHub Pull Request #11831