Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-17444

StreamingFileSink Azure HadoopRecoverableWriter class missing.

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Auto Closed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: FileSystems
    • Labels:

      Description

      Hello, I was recently attempting to use the Streaming File Sink to store data to Azure and get an exception error that it is missing the HadoopRecoverableWriter. When I searched if anyone else had the issue I came across this post here on [Stack Overflow |https://stackoverflow.com/questions/61246683/flink-streamingfilesink-on-azure-blob-storage]. Seeing no one responded, I asked about it on the mailing list and was told to submit the issue here.

      This is exception message they posted below but the stack overflow post goes into more details of where they believe the issue comes from. 

      java.lang.NoClassDefFoundError: org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter                                                            
          at org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)                                         
          at org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.createRecoverableWriter(PluginFileSystemFactory.java:129)              
          at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)                                     
          at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:117)                                                           
          at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:288)               
          at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:402)                              
          at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)                                
          at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)                              
          at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)                               
          at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:284)                                    
          at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)                                                    
          at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)                                                      
          at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
          at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)                                                               
          at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)                                                                     
          at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)                                                                                      
          at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)                                                                                        
          at java.lang.Thread.run(Thread.java:748)                          
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                bigmarie7 Kasia
              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: