2.2.2, 2.2.3, 2.3.3, 2.4.0
OS: Ubuntu Linux 18.10
Java: java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
Reproducible with a single-node Spark in standalone mode.
Reproducible with Zepellin or Spark shell.
Steps to reproduce:
The last line fails with:
The problem happens as well when checkpointing directory is placed on the local file system, on a single-node setup.
Debugging at the FS level showed that the driver requests to recursively delete the checkpointed rdd-51 soon before the exception gets thrown by the task.