[FLINK-11085] NoClassDefFoundError in presto-s3 filesystem

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.7.1, 1.8.0
• Component/s: FileSystems
• Labels: pull-request-available
• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 06/Dec/18 13:17
• Updated: 02/Oct/19 17:47
• Resolved: 12/Dec/18 15:00

Description

A user has reporter an issue on the ML where using the presto-s3 filesystem fails with an exception due to a missing class. The missing class is indeed filtered out in the shade-plugin configuration.

java.lang.NoClassDefFoundError: org/apache/flink/fs/s3presto/shaded/com/facebook/presto/hadoop/HadoopFileStatus
	at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.directory(PrestoS3FileSystem.java:446)
	at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.delete(PrestoS3FileSystem.java:423)
	at org.apache.flink.fs.s3.common.hadoop.HadoopFileSystem.delete(HadoopFileSystem.java:147)
	at org.apache.flink.runtime.state.filesystem.FileStateHandle.discardState(FileStateHandle.java:80)
	at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.doDiscard(CompletedCheckpoint.java:250)
	at org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:219)
	at org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore.addCheckpoint(StandaloneCompletedCheckpointStore.java:72)
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:844)
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:756)
	at org.apache.flink.runtime.jobmaster.JobMaster.lambda$acknowledgeCheckpoint$8(JobMaster.java:680)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Issue Links

• is caused by: MSHADE-305 Filters are merged by index, not by artifactId
• is caused by: FLINK-10366 Create a shared base for S3 file systems
• is related to: FLINK-11100 Presto S3 FileSystem E2E test broken
• links to: GitHub Pull Request #7261