Major Description
2022-03-21 17:19:02,498 ERROR org.apache.hudi.sink.compact.CompactFunction [] - Executor executes action [Execute compaction for instant 20220320025438250 from task 8] errorjava.lang.ClassCastException: org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem at org.apache.hudi.io.storage.HoodieParquetWriter.<init>(HoodieParquetWriter.java:63) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:80) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:67) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:53) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.io.HoodieWriteHandle.createNewFileWriter(HoodieWriteHandle.java:257) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:186) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:136) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.getUpdateHandle(HoodieFlinkCopyOnWriteTable.java:395) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdate(HoodieFlinkCopyOnWriteTable.java:358) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:197) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.sink.compact.CompactFunction.doCompaction(CompactFunction.java:104) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.sink.compact.CompactFunction.lambda$processElement$0(CompactFunction.java:92) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:93) ~[blob_p-1b4939f2f6d2432512518a3c63d896f6a98801a6-2e9b5fd3f311ce685d0715e6e0382fd2:0.10.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_261] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_261] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
The stack is from version 0.10.1. The root cause of the exception is:
FileSystem provided by hadoop caches FileSystem instances globally.
In flink taskmanager process, the FileSystem class is loaded by AppClassLoader,
and HoodieWrapperFileSystem is loaded by ChildFirstClassLoader.
When flink jobs recover from a failure (which is normal in k8s when the pod is exceled), ChildFirstClassLoader will change, but the AppClassLoader stay unchanged (so does the HoodieWrapperFileSystem instances cached). After recoved, when the cached instance is returned and tried to cast to a different class , the exception raises.